**Targeted In-Depth Assessment of Promising\
Practices in Secondary Vocational Education---\
What Can NAVE Do?**[^1]

**David Stern**

**University of California at Berkeley**

**May 6--7, 1999**

**Independent Advisory Panel Meeting**

**National Assessment of Vocational Education**

Summary

This paper concentrates on a strategy that uses schoolwide data to
identify secondary schools where students' academic achievement has been
improving at a relatively fast rate, and other schools where it has not.
Schools with the most involvement in vocational education can be
selected from the group where achievement is increasing relatively fast,
and from the group where it is not. These two sets of schools can then
be compared in depth, to describe how vocational education is being used
in each case. The in-depth study can supplement measures of academic
achievement with other measures of student attainment, including
work-related knowledge and skills. The results will answer two
questions:

-   Do secondary schools where student academic achievement is improving
    at a relatively rapid rate organize and deliver vocational education
    differently than schools where student achievement is not improving
    as fast?

-   Do secondary schools where student academic achievement is improving
    at a relatively fast rate also have higher levels of student
    attainment in work-related knowledge and skill?

Data for identifying schools could come from accountability systems that
have been developed in various states to monitor school improvement.
Another possible source of comparative schoolwide data would be networks
such as the Southern Regional Education Board's *High Schools That
Work*, or the U.S. Department of Education's New American High Schools.
On-site observations and interviews would be necessary in most cases to
describe how vocational education fits into the school's curriculum and
organization, and to document teaching methods in vocational and
nonvocational courses. This qualitative information would be used to
learn what may be distinctive about vocational education in schools
where achievement is increasing at a rapid rate. In addition,
quantitative data can be used to estimate how changes in student
achievement are related to prior changes in school practices, while
controlling for demographic characteristics of students.

This strategy for identifying promising practices in secondary
vocational education has several advantages, which are explained in this
paper:

-   The results, and the research process itself, can be directly useful
    in assisting improvement efforts of schools, districts, and
    states----as well as helping Congress identify effective uses of
    vocational education.

-   Starting with schools that have shown rapid improvement in student
    academic achievement is consistent with the emphasis in the 1998
    Perkins Act on developing students' academic as well as vocational
    and technical skills, and integrating academic with vocational
    instruction.

-   Using change in student achievement, rather than the level of
    achievement, as the basis for selecting schools is consistent with
    the idea that schools can more fairly be held accountable for how
    fast students improve than for where they start.

-   The data to be used are more recent than the information in national
    longitudinal data sets.

-   Using schoolwide data can avoid statistical problems caused by
    certain kinds of selection bias.

-   Comparing schools is more feasible on a large scale than conducting
    studies that depend on random assignment of students to different
    treatments.

Why Use Schoolwide Data?

Some high schools with substantial involvement in vocational education
are making rapid gains in student performance, while others are not.
Why? One of the central questions for NAVE is to understand how
vocational education is being used to advance students' academic and
work-related achievement.

To assess the effects of federally funded vocational education on the
academic performance and labor market success of high school students,
three main kinds of data are available. Advantages and disadvantages of
each type of data are reviewed briefly here. The conclusion is that,
given the constraints on NAVE's time and resources, the best way to gain
insight into promising practices in high school vocational education is
to start with available quantitative data on entire high schools, and
select certain schools for in-depth qualitative study.

**Longitudinal data on individual students**. First, longitudinal data
on individual students from nationally representative surveys such as
High School and Beyond (HSB), the National Educational Longitudinal
Survey (NELS), and the National Longitudinal Survey of Youth (NLSY) can
be used to estimate associations between certain aspects of students'
high school experience and their subsequent success at school or work,
controlling for students' background characteristics. The main advantage
of these surveys is that they comprise large samples of students who
represent the national population, and are therefore amenable to
statistical analysis. Much of the published research on effects of
vocational education has relied on these datasets.

Several reviews of this research have been conducted (Bishop 1989;
Boesel et al. 1994; Stern et al. 1995). They point to improvement in the
research resulting from the availability of transcript data showing the
actual courses taken by students while in high school. Using such data,
a number of studies have found that high school vocational courses are
more likely to lead to regular employment and high wages when graduates
work in jobs where they can use the specific occupational knowledge and
skills taught in those courses. On the other hand, students who take
more vocational courses in high school are usually found to be less
likely to enroll in postsecondary education. These findings are not very
surprising.

Another finding that emerges from several studies that use national
longitudinal data sets is less obvious. High school students who combine
a substantial academic curriculum with several vocational courses have
been found to do better than students who omit either one of these two
components. This was first noted by Kang and Bishop (1989), who
discovered a positive interaction in the HSB data between the number of
academic courses and the number of vocational courses in predicting
post-high school earnings for males who did not attend college. Arum and
Shavit (1995), also using the HSB data, used transcripts to identify
students who had taken a set of advanced academic courses, a sequence of
vocational courses, or both. Four years after senior year, individuals
who had combined academic and vocational courses in high school had the
greatest likelihood of being employed in professional, managerial, or
skilled jobs, or being enrolled in postsecondary education. Outcomes for
this group were better, or at least as good, as those who had taken
advanced academic courses but no vocational education. Levesque, Lauen,
Teitelbaum, Alt, Librera, Joyner, and Nelson (1998), in addition to
replicating these results with HSB data (tables 61.1 and 64), also found
in the NELS data that students who combined a college-preparatory
academic curriculum with a specific vocational sequence had gains in
mathematics, reading, and science test scores during high school that
were similar to the gains of students who took only the college-prep
curriculum----and both of these groups gained substantially more than
other students (tables 38--41).

The problem with these findings based on longitudinal surveys is that
they do not prove anything about cause and effect. The analysis can
never fully eliminate the possibility of selection bias----that some
unmeasured attributes of students account for both their participation
in certain programs and their subsequent performance. For instance, the
reason why students who are employed in jobs that use their specific
occupational training earn higher wages may be that they enter high
school already possessing specific abilities or interests which cause
them to choose certain courses where they can excel, and also cause them
to perform well on jobs that tap those same interests or abilities. The
occupational courses themselves may contribute little or nothing to the
abilities and interests that make these people valuable to employers.

Similarly, the reason why students who take more vocational courses are
less likely to enroll in postsecondary education may be that they come
to high school less interested in the kind of book learning that schools
emphasize, so they choose more hands-on courses while in high school,
and are also more likeley to stop their formal schooling as soon as they
graduate. The vocational courses themselves may do nothing to deter them
from going on to college. Finally, the reason why students who combine a
strong academic curriculum with an occupational sequence perform better
both at school and at work may be that they start high school already
possessing more ambition, energy, self-discipline, or awareness of what
it takes to do well in the world, which causes them to take a doubly
demanding curriculum in high school and also to excel in their post-high
school pursuits. Their built-in get-up-and-go might enable them to
succeed just as well without the extra coursework.

Social scientists have tried hard to find ways of finessing the
selection problem. One of the most sophisticated attempts is Altonji's
1995 analysis of data from the National Longitudinal Survey of the High
School Class of 1972, which he boldly describes as "the first systematic
study of the effects of secondary school curriculum on postsecondary
education and on success in the labor market" (p. 410). Altonji used the
average number of courses in each subject taken by all students at a
high school as an instrumental variable to predict the number of courses
in that subject taken by each student at that high school. This was
intended to reduce or eliminate the endogenous component of
course-taking by individual students, which is what creates selection
bias. He also compared the results with ordinary least-squares
regressions. The results using instrumental variables often differed
from those using ordinary least-squares, which may indicate that
selection bias really is a problem.

Surprisingly, foreign language courses stood out in Altonji's results as
the best predictor of both post-high school wages and college attendance
(tables 2 and 3). No other academic or vocational subject had a
consistently significant association with wages. Altonji grouped
vocational courses into "industrial" and "commercial"; both were
negatively associated with college attendance. Altonji does not accept
these results at face value (though the Modern Languages Association
might be pleased if he did!). He surmises that foreign language courses
may contribute to "general skill development," or else the result may be
due to omitted variables or sampling error. He is puzzled by the small
estimated effects of coursework in other subjects, particularly
mathematics and science. Unfortunately, despite the attempt to rid the
data of selection bias, the findings and discussion do not inspire much
confidence in the possibility of reaching clear conclusions about cause
and effect.

A final problem with the national longitudinal data sets is that they
captured high school experiences from the 1970s (NLS72), 1980s (HSB and
NLSY79), or early 1990s (NELS). The information therefore does not shed
any light on changes that may have occurred in high schools during the
1990s. The only new data set of this kind that might inform the current
NAVE is a new National Longitudinal Survey of Youth that began in 1997
(NLSY97). Unfortunately, the first wave of that survey did not collect
transcript information, and the sample included a predominance of
students in grades 9 and 10 who were too young to have had much
opportunity to participate in vocational or career-related instruction.

**Comparing groups of students within schools**. A second kind of data
comes from comparing certain students in a school with other students in
the same school. This can be done for more than one school at a time, if
they offer comparable programs. Generally, schools are interested in
knowing whether certain instructional practices lead to better
performance by students. But simply comparing data for different
students can be seriously misleading, again because of selection bias:
students may be given certain kinds of instruction *as a result of*
their level of performance.

Consider a hypothetical example in which students who report that they
make oral presentations in science class also obtain higher scores on a
science achievement test. One explanation could be that high-achieving
students are asked to make oral presentations because their teachers
think they will set a good example for their classmates. Or an entire
class in advanced science might be asked to do oral presentations
because they have mastered the basics and the teacher believes they are
ready for more individual work. In other words, being asked to do oral
presentations could be the result, not the cause, of higher achievement.
This is another example of selection bias. Consequently, the observed
correlation does not imply that assigning more oral presentations for
low-achieving students would necessarily improve their performance.

It would be different if a school found that the *average* science score
for students in the whole school improved after science teachers for
some reason decided to begin assigning oral presentations to a *larger
proportion* of students. This finding, based on change scores at the
school level, would provide more compelling support for extension of the
practice to additional students. This illustrates the benefit of using
schoolwide data, as explained below.

Examples of actual studies that have compared groups of students within
schools include some evaluations of career academies (Stern, Dayton, and
Raby 1998). A career academy is a small learning community in which a
group of students take classes together from the same set of teachers
for a period of two to four years. The curriculum combines a
college-preparatory academic core with vocational courses related to the
theme of the academy----for example, health careers, finance, or
electronics. Partnerships with employers and other community groups
provide internships and other connections between high school and the
outside world.

Three separate evaluations of career academies in California have
compared the performance of academy students with similar students in
the same school. That is, each academy student is individually matched
with a nonacademy student of the same gender and race, whose prior
academic record (grades, attendance, test scores, disciplinary
infractions) is as similar as possible. These evaluations have followed
both sets of students over time, and have consistently found that
academy students attend school more regularly, earn higher grades,
complete more courses, and are more likely to finish high school.
However, it is possible that these results are affected by selection
bias. If students who take the initiative of applying to a career
academy have more of some unobserved positive characteristic such as
ambition or planfulness, this could account for some or all of the
difference in performance between the two groups. On the other hand, if
academies tend to select students who are more troubled or recalcitrant,
the observed results would underestimate the true effect of academies.
The possibility of selection bias therefore makes the interpretation of
results somewhat uncertain.

A similar uncertainty occurs in evaluations which evaluate new forms of
vocational education by comparing the performance of successive
graduating classes of vocational "completers." If improvements in the
vocational curriculum over time result in higher achieving students
being attracted to vocational classes, this selection effect could cause
the measured performance of vocational completers to improve, even if
there were no improvement in the average performance of students
schoolwide. The best way to tell whether this is really happening would
be to collect data for the school as a whole.

The standard procedure to protect against selection bias is random
assignment of students to the program and control groups. If students
are randomly assigned, unobserved characteristics such as ambition or
ability to plan or recalcitrance are unlikely to differ very much on
average between the program and control groups. Random assignment has
often been done in studies of educational practices at the level of the
classroom, but it is more difficult to conduct a random assignment study
of programs that span more than one classroom. Still, such studies have
also been done despite the difficulty and expense. In particular, an
ongoing random assignment study of career academies has found, among
other things, that students in academies say they receive more support
from teachers and peers, and find schoolwork more interesting and
relevant, compared with nonacademy students (Kemple 1997). This
evaluation also has found that career academy students participate more
in career awareness activities and work-based learning. Among students
whose prior performance in school was especially weak, academies also
reduce the probability of dropping out.

Even when random assignment can be done, however, the results may not be
definitive. For example, suppose a high school starts a new program and
invites students to apply. From among the applicant pool, some students
are selected at random for the program, and the others become the
control group. Suppose that both groups are doing well a few years
later, but there is no significant difference between their results. A
possible explanation could be that the students who applied for the
program all had relatively high motivation to succeed, so they all did
well whether they were admitted to the program or not. It is also
possible that highly motivated students still need well-structured
learning opportunities in order to succeed, and that students in the
control group were able to find other such opportunities in the school.
But if the number of well-structured learning opportunities available in
the school is less than the number of potentially motivated students,
and if the new program increased the number of opportunities, then
eliminating the program would lead to poorer results, since some
students in the program or in the control group would be unable to find
good learning opportunities. This kind of effect could be detected only
by using data for the school as a whole.

**Comparing rates of change between schools**. The third option,
collecting schoolwide data, would permit analysis at the school level.
This is consistent with many current school reform efforts which
emphasize the school site as the locus for planning, implementing, and
monitoring improvements (e.g., Coalition of Essential Schools, *HSTW*,
Obey-Porter programs). Most high schools are involved in a number of
improvement efforts at any given time, with new initiatives happening
constantly. In this environment, it is difficult to isolate the effect
of any particular initiative because conditions for students not
participating in that initiative are also likely to be changing.
Nevertheless, it is possible for a school to learn from schoolwide data
whether student performance is improving over time, and whether the rate
of improvement is faster or slower than for other schools. Given that
information, schools can attempt to explain why. The National Center for
Research in Vocational Education (NCRVE) has engaged a number of high
schools in a cycle of inquiry through which schools compare data in
order to learn from their own and one another's experience.[^2]
Collecting and analyzing data on whole schools therefore can feed
directly into the schools' own self-improvement efforts. This is an
important advantage of such data.

Using schoolwide data is consistent with the emphasis in the Perkins Act
on developing students' academic as well as vocational and technical
skills, and integrating academic with vocational instruction. Federal
law in 1990 and 1998 views vocational education as an integral part of
the high school program. In fact, a review of transcripts for a national
sample of 1994 high school graduates found that 97.2 percent of public
high school students had taken at least one vocational course, and 90.8
percent had completed at least one specific occupational[^3] course
(Levesque et al. 1998, p. 12). The average high school graduate in 1994
had completed nearly four Carnegie units in vocational education courses
(Levesque et al. 1998, p. 37).

Moreover, many students who take vocational courses go to college after
graduating from high school. College-going students from the graduating
class of 1982 accounted for 48 percent of all occupationally specific
vocational coursework taken by members of that class during their four
years of high school (Muraskin 1993, p. 101). The number of students who
combine vocational education with college-preparatory academic
coursework has grown during the 1980s and 1990s. Among high school
graduates who had taken three or more specific occupational courses, the
proportion who also completed a course in chemistry rose from 15 percent
in 1982 to 34.6 percent in 1994, and the proportion who took physics
grew from 7.8 to 13 percent (Levesque et al. 1998, table 36.1).

In the accountability provisions of the 1998 Perkins Act, Congress did
not restrict states to measuring outcomes only for students
participating in vocational education. Likewise, NAVE's studies of
promising practices should not be limited to some subset of students who
are defined as "vocational." Such a definition would have to be
arbitrary and artificial, given the fact that almost all students take
at least one vocational course. Identifying a group of high schoolers as
"vocational students" would make no more sense than labeling some as
"mathematics students," since virtually all students also take at least
one mathematics class. It certainly would be possible to separate out
students who took a relatively large number of vocational courses, just
as it would be possible to distinguish those who took a lot of
mathematics. But this kind of distinction is unnecessary and irrelevant
if NAVE's purpose is to discover how vocational education is being used
in conjunction with academic instruction to improve the performance of
any or all students along a number of dimensions, including academic
achievement and preparation for postsecondary education as well as
work-related knowledge and skill.

In deciding not to require states to identify a set of vocational
students for purposes of accountability, Congress continued a strategy
it adopted in the 1994 Improving America's Schools Act (IASA). For
nearly 30 years, Title I of the Elementary and Secondary Education Act
had distributed federal money to low-income schools for compensatory
education, with accountability provisions stipulating that the money
could be spent only on students who were identified as meeting specific
criteria. In addition to being cumbersome, the attempt to trace money to
specific students led to those students being overtly labeled,
segregated, and sometimes stigmatized. In 1994 this approach was
replaced by IASA's new emphasis on schoolwide improvement. Although the
1998 Perkins Act does not explicitly embrace the schoolwide strategy of
IASA, it does avoid a definition of accountability that depends on
identifying a subgroup of students.

Given a focus on the whole school, another important principle is that
schools should be compared in terms of *change*, not just the absolute
level of student performance. Comparing schools in terms of absolute
levels of performance can make some schools look bad just because their
students come from low-income neighborhoods where educational
performance tends to be lower than in more affluent places. Comparing
change over time avoids this problem, creating a more level playing
field----although shifts in student populations still may have to be
taken into account in explaining why student achievement is improving
faster in some schools than in others.

Where to Find Schoolwide Data?

The kind of schoolwide data needed would permit analysis of the trend
over time in students' performance, and comparison with other schools.
At a minimum, it would suffice to measure performance of a simple random
sample of students at one or more grade levels, e.g., grade 12, and to
repeat the same measurement from one year to the next. To ensure that
changes are not entirely due to shifts in the composition of students,
it would also be important to collect demographic data on the students
whose performance is measured. Performance measures could include those
specified in the accountability section of the Perkins Act: academic and
technical skills attained, diplomas completed, postsecondary placements,
and participation in nontraditional fields.

One likely source of this data would be individual states that have
established accountability systems of their own. A number of states now
collect data on individual schools over time, which permit comparisons
among schools within a given state. According to *Education* *Week*
(1999, pp. 85--87), the following 13 states maintained web sites that
contain "report cards" on individual schools, with test data for more
than one year in subjects that include English or mathematics at grade
11. In the following list, an asterisk denotes states where the tests
include some performance measures in addition to multiple-choice tests:

Alabama: English/language arts, history/social studies, mathematics,
science

Arizona: English/language arts, mathematics

Florida: English/language arts, mathematics

Georgia: English/language arts, history/social studies, mathematics,
science

Kentucky\*: History/social studies, mathematics, science

Michigan\*: English/language arts, history/social studies, mathematics,
science

Nevada: English/language arts, mathematics

New Jersey\*: English/language arts, mathematics

New York\*: English/language arts, history/social studies, mathematics,
science

Ohio[^4]: English/language arts, history/social studies, mathematics,
science

Pennsylvania\*: English/language arts, mathematics

Washington: English/language arts, mathematics

West Virginia[^5]: English/language arts, history/social studies,
mathematics, science.

Ten other states test 11th graders in subjects that include English or
mathematics, but either do not include school report cards on their web
sites or else do not report data for more than one year. These are:

California: English/language arts, history/social studies, mathematics,
science

Idaho: English/language arts, mathematics

Maine\*: English/language arts, history/social studies, mathematics,
science

Mississippi: English/language arts, mathematics

Missouri\*: English/language arts, history/social studies

Montana: English/language arts, history/social studies, mathematics,
science

Oklahoma: English/language arts, history/social studies, mathematics,
science

South Dakota: English/language arts, history/social studies,
mathematics, science

Utah: English/language arts, history/social studies, mathematics,
science

Wyoming\*: English/language arts, mathematics.

A particularly good source of data on the contribution of vocational
education to school improvement is the multistate network called *High
Schols That Work (HSTW)*. This is the nation's first and largest effort
to combine challenging academic courses and modern vocational
educational studies to raise the achievement of high school students who
were not enrolled in college-prep courses. As of 1999, the *HSTW*
network includes close to 800 schools. The two major goals of the
initiative are: (1) to raise the mathematics, science, communication,
problem-solving and technical achievement of career-bound youth to the
national average; and (2) to blend the essential content of traditional
college preparatory studies----mathematics, science, language arts and
social studies----with quality vocational and technical studies (Bottoms
and Presson 1995; Southern Regional Education Board 1996).

One of the key features of *HSTW* is the biennial assessment of student
progress. Every two years, seniors who are classified as "vocational
completers" take a special *HSTW* achievement test, based on the
National Assessment of Educational Progress tests in mathematics,
language, and science. Data from this test, along with results of a
student survey, faculty survey, and follow-up of recent graduates, are
compiled by the Educational Testing Service in a report to each school.
Site visits from *HSTW* staff are occasions to examine school practices
in the light of this data. The intent is to engender continuous
improvement aided by periodic data collection. Unlike the data routinely
collected by state accountability systems, the *HSTW* assessment
includes information about vocational and academic courses taken by each
student, and reports by students about specific instructional practices
relating academic concepts to vocational applications.

Beginning in 1998, some *HSTW* schools started collecting assessment
data on schoolwide samples of students, instead of vocational completers
only. Twenty-seven *HSTW* sites located in 20 different states[^6]
collected 1998 data from a fully representative sample of the senior
class. Repeating this sampling procedure in the next assessment,
scheduled for January-February 2000, would make it possible to compare
schoolwide progress among these 27 high schools. Data would become
available in the spring of 2000, making it possible to identify schools
with relatively fast and slow rates of improvement in student
achievement. On-site studies could then be conducted during the
2000-2001 school year, to determine how vocational education is being
used in schools with rapid rates of improvement compared to other
schools. This would allow time to analyze the qualitative and
quantitative information and write up the results before the NAVE
interim report is due at the end of 2001.

The decision by these schools to repeat the schoolwide sampling
procedure in 2000 will have to be made by August 1999, because it takes
several months to get ready for the assessment. Schoolwide sampling in
these schools entails an additional cost beyond what *HSTW* normally
provides for collecting data on vocational completers. In 1998 the cost
of testing the additional students was approximately \$65,000, paid by
NCRVE as part of its research study on using comparative data for school
improvement. To ensure the continuation of this unique source of
schoolwide data relating student achievement to vocational education,
NAVE might have to provide all or part of the additional cost in 2000.

Another possible source of data on schools that include vocational or
work-related education as a central component of their curriculum is the
set of New American High Schools identified by the Office of Vocational
and Adult Education (OVAE) in the U.S. Department of Education. OVAE is
asking these schools to provide trend data on one or more performance
indicators in the following categories:

Academic performance: Advanced Placement and SAT scores, other
achievement tests, percentage of graduating seniors meeting state
university's entrance requirements, or percentage of senior
portfolios/exhibitions/projects achieving satisfactory or higher rating;

Technical or work-readiness skills: percentage of seniors in work-based
learning who receive a satisfactory or higher rating by employer;

High school graduation rate;

Completion of a career program or major;

Dropout rate: four-year completion rate, or annual dropout rate;

Attendance: average days absent, or average daily attendance;

Discipline problems: number of reported episodes, or number of
suspensions/expulsions.

It should be possible for the NAVE staff, assisted by the Advisory Panel
and contractors, to determine fairly quickly which state accountability
systems can provide useful data, along with *HSTW* and New American High
Schools. There may also be other data sets that meet the desired
criteria. Possibilities include the Beacon schools recognized by the
Annenberg Foundation, schools that have won awards from the Association
for Career and Technical Education, or schools participating in the
International Skills Olympics for vocational students.

How to Analyze Schoolwide Data?

The main strategy proposed here is to conduct qualitative studies within
a quantitative frame. That is, numerical data would be used to identify
schools that show relatively rapid rates of improvement over time.
Descriptive information from observations and interviews would then
provide insights on how vocational education is delivered in these
schools, compared to others. The result will be a demonstration of how
vocational education can be part of an instructional package in schools
where student attainment is growing relatively fast.

**Qualitative studies within a quantitative frame**. The minimum
necessary quantitative data for this analysis would consist of
information on multiyear trends in student performance as measured by
average scores on academic achievement tests. In addition to data on
average performance (mean or median), it would be highly desirable also
to measure performance in the high and low portions of the distribution
(using quartile scores or standard deviations), so that schools can
monitor trends across the whole spectrum of student achievement. Along
with standardized tests of academic achievement, measures of
work-related knowledge or skill, or at least application-oriented tests
of academic achievement, would be especially pertinent to measuring the
impact of vocational education.[^7] In addition to test scores, other
important indicators of student performance would include dropout and
graduation rates, and postsecondary placements in work or further
education. Follow-up data at least a year after graduation would also be
revealing, but few schools or states collect such information.

Given such indicators, analysis would begin by ranking schools according
to how fast they are improving over time. The total number of schools to
be identified for in-depth study will depend on the total budget
available and the amount of qualitative detail desired. A plausible
scenario would be to choose schools as follows:

\(1\) Select three to five states where the accountability systems
provide the necessary data on schools. In each of these states, identify
three to five schools where student achievement has improved at a rapid
rate, and two or three schools where it has not. This would result in
selection of between 15 and 40 schools for direct, in-depth study.

\(2\) From the 27 schools in *HSTW* with schoolwide data, select three
to five schools with the fastest rates of growth in student achievement
between 1998 and 2000. Also choose two or three where student
achievement has not grown as rapidly. This would add between five and
eight schools.

\(3\) From the New American High Schools, choose another five to eight
schools using the same procedure.

These three sets add up to between 25 and 56 schools. If \$5,000 were
allocated to pay for visiting each of these schools and writing
descriptions based on observations and interviews, the total cost would
be between \$125,000 and \$280,000.

Since the central purpose of this study is to understand the role of
vocational education, it is necessary to ensure that vocational
education has a substantial presence in each of the schools to be
studied. Schools in the *HSTW* project may be presumed to satisfy this
criterion, because vocational education plays an important part in the
key practices promoted by this project. Similarly, attention to
career-related instruction is one of the characteristics of the New
American High Schools.

However, in choosing schools based on data from state accountability
systems, it will be necessary to verify involvement in vocational
education before sending out teams to conduct in-depth studies. For
instance, if the aim in a particular state is to select five schools
where student achievement has grown rapidly and three where it has not,
it may be necessary to start with a preliminary set of ten or fifteen
fast-improving schools and six or nine slow-improving schools. From this
preliminary set, schools for in-depth study can be chosen based on
involvement in vocational education. Indicators of involvement in
vocational education might include the average number of vocational
courses students have taken, or the amount of money budgeted per pupil
for vocational instruction. Information on vocational involvement may be
obtained from the state education agency, local district offices, or
preliminary telephone calls to the schools themselves.

Once schools have been identified based on rate of improvement in
student achievement and degree of involvement in vocational education,
they can be visited by NAVE staff or contractors to conduct observations
and interviews on the role of vocational education. The first question
to be answered is whether and how schools where student academic
achievement is improving at a relatively rapid rate organize and deliver
vocational education differently than schools where student achievement
is not improving as fast. Observations and interviews should look for
differences along the following dimensions, among others:

-   Integrating academic and vocational curriculum and instruction. Do
    all or most students take a set of challenging academic classes as
    well as a coherent sequence of vocational courses? Do vocational
    courses reinforce academic skills and concepts? Do academic courses
    include work-related applications? Do students engage in projects
    that link academic and vocational subjects?

-   Work-based learning. Do all or most students engage in job
    shadowing, service learning, internships, school enterprises, or
    other forms of work-based learning? Are these experiences tied
    directly to academic courses? Do nonvocational teachers participate
    in work experiences outside of school, and do they supervise
    students who do the same?

-   Connecting secondary with postsecondary education. Do all or most
    students satisfy prerequisites for admission to a four-year college
    or university? Do they acquire occupational skills that will make it
    easier for them to work their way through college? Are there
    explicit arrangements for students to earn college credits while in
    high school?

In addition, a second important question is whether schools where
student academic achievement is improving at a relatively fast rate also
have higher levels of student attainment in work-related knowledge and
skill. State accountability systems seldom if ever include measures of
work-related knowledge and skill, so whatever data are available will
have to be collected as part of the in-depth study of each school.
Bishop (1999) has proposed a set of possible measures. Along with
measures of students' actual knowledge and skill, it will also be useful
to describe dimensions such as:

-   Career-related curriculum. Does the high school offer a set of
    curricular themes related to broad industry groupings or
    occupational clusters? Do all or most students choose such a theme?
    Do students share their core academic classes (English, mathematics,
    social studies, science) with others who have chosen the same theme?

-   Postsecondary pathways. Do curricular themes in high school lead
    directly to opportunities for employment and further education? Do
    students receive any kind of formal certification along the way?

The core of the strategy proposed here would use descriptive information
about these kinds of dimensions to compare schools where student
performance is improving rapidly with schools where it is not. If
schools with fast growth in student achievement organize and deliver
vocational education differently than other schools, it will be possible
to conclude at least that rapid gains in student attainment can occur
when certain kinds of vocational education are practiced. The results
will not prove that observed differences in programs or practices caused
the differences in student performance, since it is conceivable that
rapid growth in student performance is what permits these practices to
be adopted. This can be debated. But it will still be useful to Congress
and the field simply to know whether schools with rapid improvement are
practicing a different kind of vocational education.

Specific data collection procedures for the in-depth, on-site studies
will not be detailed here, but there are several closely related
precedents. Bradby and Teitelbaum (1998) piloted this kind of
qualitative study within a quantitative frame for NCRVE, using data from
*HSTW*. Rock and Ham (1999) describe a similar approach being used to
evaluate Accelerated Schools. Hudis and Visher (1999) give descriptions
of New American High Schools which show how qualitative accounts can
compare certain practices among schools. Stasz (1999) outlines how
qualitative studies of high schools can be conducted for NAVE.

**Purely quantitative studies**. With aggegrated data from a few dozen
schools, it is also possible to conduct purely quantitative analysis.
Bloom (1999) has explained how to use trend data to test whether a
particular intervention has had a significant impact on a set of
schools. Under plausible assumptions, a set of 20 schools would be
sufficient to detect an effect size of only 0.2 standard deviation. The
virtue of this method is that it supports fairly strong inferences about
cause and effect, based on changes in student performance after the
intervention. However, Bloom's procedure assumes that schools are
adopting a well-defined new program or practice, beginning at a
particular point in time. In most high schools, new initiatives are
emerging all the time, so Bloom's method may not be appropriate.

Nevertheless, aggregate data from a set of schools over time may be used
to detect whether changes in school practices are associated with
subsequent changes in student performance. Standard regression analysis
can be applied here if the number of schools exceeds 30, and preferably
more. Changes in student body characteristics can also be taken into
account. Given NAVE's two-year timeline, this kind of analysis would
require retrospective data on school practices.

Additional kinds of analysis can be conducted if data are available on
individual students, including not only measures of their performance
and background characteristics but also course-taking data and other
indicators of their educational experience. For example, *HSTW* collect
such information in their biennial assessment of seniors. Statistical
techniques like hierarchical linear modeling can be used to measure the
degree of association between students' individual educational histories
and their performance on tests, as well as how those associations depend
on school characteristics. For instance, the association between test
scores and whether a student has taken a sequence of vocational courses
may depend on whether most students in the school take such a vocational
sequence, or only a few. NAVE may want to commission a special analysis
of the unique *HSTW* data set to answer questions such as this.

References

Altonji, J.G. 1995. "The Effects of High School Curriculum on Education
and Labor Market Outcomes." *Journal of Human Resources* 30 (3):
409--439.

Arum, R. and Shavit, Y. 1995. "Secondary Vocational Education and the
Transition From School to Work." *Sociology of Education* 68: 187--204.

Bishop, J. 1989. "Occupational Training in High School: When Does It Pay
Off?" *Economics of Education Review* 8 (1): 1--15.

Bishop, J. 1999. "Methods of Measuring the Effectiveness of Alternative
Ways of Delivering Vocational Education." Design paper prepared for the
NAVE Advisory Panel, Cornell University, Ithaca, NY.

Bloom, H.S. 1999. "Estimating Program Impacts on Student Achievement
Using 'Short' Interrupted Time-Series." Paper presented at the meeting
of the American Educational Research Association, Manpower Demonstration
Research Corporation, New York.

Boesel, D., Hudson, L., Deich, S., and Masten, C. 1994. *National
Assessment of Vocational Education, Final Report to Congress, Volume II:
Participation in and Quality of Vocational Education*. Washington, D.C.:
U.S. Department of Education, Office of Educational Research and
Improvement.

Bottoms, G. and Presson, A. 1995. "Improving High Schools for
Career-Bound Youth," in *Education Through Occupations*, 35--54. Ed.
W.N. Grubb. New York, NY: Teachers College Press.

Bradby, D. and Teitelbaum, P. 1998 (August Draft). *Practice Makes
Perfect:* HSTW *Practices and Increased Student Achievement*. Berkeley,
CA: MPR Associates.

Hudis, P.M. and Visher, M.G. 1999 (March Draft). *Aiming High:
Strategies to Promote High Standards*. Berkeley, CA: MPR Associates.

Kang, S. and Bishop, J. 1989. "Vocational and Academic Education in High
School: Complements or Substitutes?" *Economics of Education Review* 8
(2): 133--148.

Kemple, J.J. 1997. *Career Academies: Communities of Support for
Students and Teachers: Emerging Findings From a 10-Site Evaluation*. New
York: Manpower Demonstration Research Corporation.

Levesque, K., Lauen, D., Teitelbaum, P., Alt, M., Librera, S., Joyner,
N., and Nelson, D. 1998 (November Draft). *Vocational Education in the
United States: Toward the Year 2000*. Washington, D.C.: U.S. Department
of Education, National Center for Education Statistics.

Muraskin, L.D. 1993. *Secondary Vocational Education: Availability,
Coursetaking, and Outcomes*. Rockville, MD: Westat, Inc.

"Quality Counts '99: Rewarding Results, Punishing Failure." *Education
Week*. 1999: 85--87.

Rock, J. and Ham, S. 1999. "Evaluating Whole-School Reforms Using
Quantitative and Qualitative Research Methods." Paper presented at the
meeting of the American Educational Research Association, Manpower
Demonstration Research Corporation, New York.

Southern Regional Education Board. 1996. *High Schools That Work:* High
Schools That Work *Blends Academic and Vocational Education to Raise
Student Achievement*. Atlanta, GA: Author.

Stasz, C. 1999. "Assessing the Quality of Vocational Education in High
Schools." Design paper prepared for the NAVE Advisory Panel, Santa
Monica, CA: RAND.

Stern, D., Dayton, C., and Raby, M. 1998. *Career Academies and High
School Reform*. Berkeley, CA: Career Academy Support Network, University
of California.

Stern, D., Finkelstein, N., Stone, J.R. III, Latting, J., and Dornsife,
C. 1995. *School to Work: Research on Programs in the United States.*
Washington, D.C. and London, England: Taylor and Francis, Falmer Press.

[^1]: The views expressed here are the author's alone and should not be
    attributed to the sponsoring agency.

[^2]: The schools involved include 27 from *High Schools That Work*,
    described below, and 5 from the Bay Area School Reform
    Collaborative. A report on this actitivity will be published by
    NCRVE at the end of 1999.

[^3]: Specific occupational courses exclude keyboarding, work
    experience, family and consumer sciences, industrial arts or
    technology courses.

[^4]: Ohio tests at grade 12, not grade 11.

[^5]: West Virginia also tests at grade 12, in addition to grade 11.

[^6]: Alabama, Arkansas, Delaware, Florida, Georgia, Hawaii, Indiana,
    Kansas, Kentucky, Louisiana, Massachusetts, Maryland, North
    Carolina, Oklahoma, Pennsylvania, South Carolina, Texas, Virginia,
    Washington and West Virginia.

[^7]: For example, the Work Keys assessment system has been developed by
    ACT to measure employability skills. It consists of tests lasting 40
    to 50 minutes in each of eight areas: applied mathematics, applied
    technology, listening, locating information, observation, reading
    for information, teamwork, and writing. Some of these tests are
    intended to assess individuals' ability to apply academic skills,
    for example in mathematics and writing. However, even at the highest
    levels of difficulty, the test items are not designed to measure
    detailed or advanced knowledge in any particular field of work.
    Examples can be seen at http://www.act.org.