NATIONAL ASSESSMENT OF VOCATIONAL EDUCATION
(NAVE)![](media/image1.png){width="1.3902777777777777in"
height="1.1333333333333333in"}

######## OVERVIEW OF EVALUATION PLAN {#overview-of-evaluation-plan .NAVE-Subtitle}

# I. INTRODUCTION

Federal support for vocational education, and for understanding its
consequences, has a long history. As was true with previous vocational
legislation, the recently amended (1998) Carl Perkins Vocational and
Technology Education Act (Perkins III), directs the Secretary of
Education to complete an "independent evaluation and assessment of
vocational and technical education programs under this Act," in this
case, by 2002. As currently designed, this national assessment plan will
include multiple components. This paper discusses the main issues that
NAVE will address, and the research strategies that will be undertaken
to address those issues.

Perkins III (section 114) directs the NAVE to address a wide range of
topics, including: (1) implementation of state and local programs; (2)
impact of changes in federal funding formulas;\
(3) teacher quality and teacher supply and demand; (4) student
participation in vocational education, (5) academic and employment
outcomes; (6) employer involvement and satisfaction with vocation
education programs; (7) education technology and distance learning; and
(8) the impact of accountability requirements on program performance.
Perkins III also directs the Secretary to appoint an Independent
Advisory Panel to provide advice on conducting the NAVE and to submit to
Congress its own independent analysis of NAVE findings and
recommendations (see Appendix A for list of panel members).

Congress has mandated a final NAVE report in 2002 to provide it with
information that can guide reauthorization of the Perkins Act in 2003.
Of primary interest is how some of the new provisions in the law have
been implemented. Unfortunately, many of the important changes in
accountability, integration with workforce development, and increased
flexibility will have barely begun. Most states have opted for
"transitional plans" which in effect, defer implementation of Perkins
III provisions until October 2000 (Figure 1). Given the July 2002 date
for reporting NAVE\
results to Congress, data collected during the 2000/2001 school year
will reflect the very early efforts made in response to specific new
provisions in Perkins III. Conclusions regarding the longer-term
prospects for Perkins III will require further research.

![](media/image2.wmf){width="6.045833333333333in"
height="1.7881944444444444in"}

At the same time, however, Perkins III continues to emphasize several
major themes reflected in the 1990 Perkins amendments. These include the
integration of academic and vocational education, broadening the focus
of vocational education to emphasize industries and careers, and
strengthening the links between secondary and postsecondary education
through tech-prep and other strategies. The current NAVE will be able to
track the extent to which these longer run themes are reflected in
school practice, and where possible, what effects they have on outcomes
for students.

All of these objectives will be addressed in a series of studies soon to
get underway. Two factors are guiding the design of the NAVE agenda: (1)
the current policy environment, and (2) the specific research questions
that reflect policymaker and practitioner interests.

A. Policy Context for the NAVE

Over the past two decades, three prior national assessments of
vocational education have been conducted.[^1] Each study sought to focus
upon what were thought to be the major issues facing vocational
education at the time---funding, special populations, and economic
competitiveness of the workforce. The current NAVE must also be
sensitive to the dominant educational issues of its time.

Vocational education is a field in transition, prompted by sweeping
changes in state and local education priorities. New goals, program
offerings, and terminology increasingly characterize vocational
education. Federal legislation has encouraged several major
changes---from an historic emphasis on entry-level job preparation in
semi-skilled occupations to a broader focus on preparation for careers
that offer high wages and requires higher level skills; from preparing
students to enter the workforce directly after high school to providing
students with the choice of pursuing employment or attending college, or
as is increasingly the case, doing both simultaneously; and from
expecting vocational students to do less well in school than other
students, to holding such students to the same academic standards as
others. Many of the overarching issues that NAVE will address will
consider whether this transition in the field of vocational education is
"on track."

Several key policy concerns are likely to shape this assessment:

-   ***Federal Funding For Vocational Education*.** Federal budgets are
    widely regarded as a basic indicator of policy priorities. Although
    overall funding on Department of Education (ED) programs has
    increased by 177 percent from FY 1980--FY 1999, vocational education
    funding increased by only 47 percent during the same period (Figure
    2).[^2] In 1980, funding for vocational education was about 6
    percent of total ED expenditures; it has now shrunk to about 3
    percent.

![](media/image3.wmf){width="6.049305555555556in"
height="3.923611111111111in"}

These trends strongly suggest that for nearly two decades vocational
education has been increasingly viewed as less worthy of investment than
other areas. One reason for this pattern may be that vocational
education has historically prepared students for work at a time when
priorities have shifted toward academic preparation for college. To the
extent that vocational education has changed to address these as well as
more traditional objectives, it will be important for NAVE to describe
and assess this development.

![](media/image4.wmf){width="3.0597222222222222in"
height="2.6006944444444446in"}

Although overall federal funding for vocational education has not
changed substantially, the Perkins Act remains the largest single source
of ED funds directed to high schools. Comparing ED's three major sources
of funding for high schools (only a share of the funds from each program
is directed at high schools), Perkins Act funds spent to help high
school students are more than Title I and TRIO funds combined (Figure
3).[^3]

Given the major role that Perkins funding plays at the high school level
relative to other ED programs, it makes sense to think about how this
investment contributes to high school improvement generally, and the
contribution that vocational education makes, or could make, to such
efforts.

-   ***Academic Reform*.** There is little question but that the
    principal focus of recent education reforms has been on improving
    students' academic achievement and increasing their opportunities to
    attend college. Federal vocational education policy now places top
    priority upon ensuring that vocational education students are
    academically well prepared for both careers and/or success in
    postsecondary education. But what is, or ought to be, the
    responsibility of vocational educators for ensuring that students
    leave high school with both a solid academic foundation as well as
    technical skills? One major goal of NAVE should be to help Congress
    better understand what the contribution of vocational education is
    to achieving these objectives and how this contribution can be
    increased.

-   ***Changes in Perkins III*.** Although Congress did not alter the
    basic structure of the Perkins Act in the 1998 reauthorization, it
    did make several important substantive changes. Among these are: (1)
    increased emphasis on academics; (2) greater flexibility in the use
    of funds through elimination of major set-asides for gender equity
    and other rules governing the use of funds; (3) a higher proportion
    of funds directed to local programs and the establishment of a 10
    percent reserve fund; and (4) creation of a "higher stakes"
    accountability system. At the same time, the Workforce Investment
    Act of 1998 (WIA) also encouraged greater integration of vocational
    education and the workforce development system. Implementing these
    changes will be a major challenge for school districts and
    postsecondary institutions.

B. Main Policy Issues

Given the educational and political context in which Perkins III
operates, as well as specific legislative changes recently enacted, the
NAVE will be guided by several main questions:

1\. What strategies improve the performance of "vocational students" and
how does, or can, vocational education contribute to improving academic
and occupational skills, access to postsecondary education, and
earnings?

Standards-based reform is fundamentally altering policies and practices
from kindergarten through high school. NAVE will examine the role of
standards---both academic and occupational---in vocational education,
the effects on students' academic and technical preparation of promising
reform strategies, and the role of federal policy, as implemented by
states and communities, in facilitating implementation of effective
programs of study.

2\. What are the pathways by which sub-baccalaureate students prepare
for careers, and what is the contribution of workforce reform efforts to
improving their training?

In passing Perkins III and the WIA, Congress was concerned that
federally supported job training programs operated independently of
vocational education. Congress expects that linking Perkins III and WIA
together will encourage better integration of vocational education and
workforce development policies. Issues to be examined include: (1) the
role of postsecondary vocational in developing state workforce training
plans; (2) the importance of unified planning; (3) the early impact on
postsecondary institutions (e.g., programs of study, one-stop career
centers, accountability) of efforts to align Perkins III and workforce
development, and (4) patterns of enrollment and participation in
postsecondary occupational education, and their relationship to outcomes
and impact.

3\. Is the policy shift from set-asides and legislative prescription to
flexibility and accountability likely to improve program quality and
student outcomes? How do special populations fare?

For the past two decades, federal policy has focused on serving those
most at-risk, commonly termed the "special populations." Perkins III
represents a major shift in direction---eliminating set-asides for
"single parents\...," requirements that local funds be prioritized to
serve the highest concentrations of special populations, and
requirements to coordinate with the Individuals for Disabilities
Education Act. In its place is an increased emphasis on accountability,
including the requirement that states track the progress of special
population groups. Has increased flexibility resulted in changes in
educational priorities or practices? Have "at-risk" populations been
helped or hurt as a result? Are accountability requirements improving
the quality of vocational education for all students?

# II. NAVE RESEARCH AGENDA 

![](media/image5.wmf){width="3.497916666666667in"
height="3.334722222222222in"}

Addressing the primary vocational policy issues requires a set of
interrelated but distinct studies. No one study could encompass the
broad array of interests and questions. Moreover, the overall research
agenda calls for diverse data collection and analysis methods,
in-cluding: qualitative case studies, national surveys, examination of
existing databases, and sophisti-cated econometric estimation. Fi-nally,
while some broad themes are relevant to both secondary and postsecondary
vocational educa-tion, each also has its own key issues.

For these reasons, the NAVE plans to undertake studies in seven main
areas (Figure 4). These are described below, grouped together by major
topic: secondary vocational education, postsecondary vocational
education, and program management.

# SECONDARY VOCATIONAL EDUCATION {#secondary-vocational-education .maintopic}

The role of vocational education at the secondary level is evolving.
Many policymakers, educators, and parents remain concerned about whether
secondary vocational programs serve students' best interests in the long
run. Previous studies suggest that completion of a vocational program
has a positive effect on employment, at least in the short run, if
students do not pursue postsecondary education[^4] and if they obtain
training-related jobs.[^5] However, the earlier research also indicates
that vocational education contributes to neither academic achievement
nor postsecondary enrollment---the path desired by most students and
their parents. Studies undertaken by the last NAVE confirmed that
vocational instruction in the early 1990s was still largely traditional,
with little focus on academic skills.[^6] These concerns about the
quality and outcomes of vocational education have contributed to the
decline in participation of high school students since the 1980s. These
same concerns have raised the stakes for gaining a clearer understanding
of: (1) how participation in and impacts of secondary vocational
education have changed since the earlier studies, (2) the extent to
which secondary vocational education now reflects high quality
practices, and (3) whether promising career education reform strategies
are proving to be effective.

A. Participation and Outcomes

Perhaps the most important issue for vocational education, and for the
current NAVE, is who participates at the secondary level and how well
they fare in school and beyond. At least through the early 1990s,
vocational education had come to be stigmatized as a high school "track"
for students with poor academic capabilities, special needs, or
behavioral problems.[^7] This result alone would not be problematic, but
vocational programs also appeared to contribute little "value added" to
student outcomes. In the current climate, with increasing federal
emphasis on program performance and on high academic achievement for all
students, vocational education has much to prove.

Over the last five years, however, career education has been given new
prominence by several initiatives targeted to students other than those
traditionally served by vocational programs. In addition, some states
and districts have worked to strengthen vocational courses.[^8] Whether
any of these efforts have successfully broadened the appeal and improved
the impact of occupational programs needs careful examination.

![](media/image6.wmf){width="4.957638888888889in" height="2.76875in"}

 Who participates in secondary vocational education?

Updating the enrollment trends and characteristics of students served by
high school-level vocational programs is of interest, particularly in
light of growing reports from state directors that enrollment patterns
have been changing. Congress and most educators, remain committed to
providing students from special populations access to vocational
education. However, many believe the quality of vocational programs is
unlikely to improve without attracting a broader segment of the student
population; or that the participation of a more diverse set of students
will signal that quality improvements are being made. Several questions
must be addressed:

-   Who enrolls in secondary vocational education and to what extent has
    this picture changed over the 1990s? How do the characteristics of
    vocational students compare with those of nonvocational students?

-   Does enrollment vary by high school setting and vocational program
    area?

-   Has the course taking (academic and technical) of vocational
    students changed?

     What is the impact of vocational education at the secondary level?

As a result of changing objectives and educational priorities, it is
important to understand the contribution of secondary vocational
education to traditional objectives (technical competency, labor market
outcomes, and general employability skills) but also to academic
achievement and postsecondary enrollment. Among the research issues are

-   To what extent does vocational education contribute to students'
    academic achievement and chances of attending and succeeding in
    college?

-   What effect does secondary vocational education have on students'
    technical preparation for work, employability skills, and ability to
    meet employers' expectations?

-   What is the impact on wages and earnings? In the short run? In the
    longer run? For noncollege-bound students? College-bound students?

-   How do these outcomes vary for different groups of students,
    particularly students from "special populations?"

B. Quality of Secondary School Vocational Education

For nearly a decade, federal policy has attempted to improve the quality
of vocational programs by strengthening the connection between
vocational education and mainstream educational objectives at the high
school level. These vocational improvements are intended to keep pace
with and complement other reform efforts in high schools. States and
local districts have been raising the academic coursework and skills
required for graduation, making high academic achievement the paramount
marker of a school's success. While other measures of school performance
are also important (e.g., placement into higher education or
career-oriented employment, reductions in drop-out rates, technical
competency), efforts to increase academic attainment are likely to
continue as a focus for school improvement. A major policy issue facing
vocational education, then, is how it can support this central mission
for high schools.

The "quality" of vocational education is clearly critical to this
objective. Perkins III builds on prior legislation in emphasizing
program improvement. While it may be too early to fully judge the
educational system's response to Perkins III, Congress will certainly be
interested in the\
extent to which actual practice is consistent with legislative and other
views of what constitutes "effective" vocational programs.

![](media/image7.wmf){width="4.95in" height="2.8777777777777778in"}

 How can schools improve the academic performance of vocational
students and what,\
if anything, is the relationship between vocational education and those
improvement\
efforts?

Federal vocational education policy now places priority on ensuring that
secondary vocational education students are academically well prepared
for careers and success in postsecondary education. For some districts
and schools, meeting these objectives requires substantial changes,
including new policies or requirements, shifts in instructional methods,
or modifications to course content; some schools have already undertaken
these reforms. While Perkins III provides guidance on program
improvement strategies, identifying the approaches used by schools that
have actually raised the academic (and technical) competence of
vocational students will be of great benefit to both policymakers and
practitioners. Several critical questions need to be addressed, such as

-   Do high schools that have succeeded in improving vocational
    students' achievement organize vocational education differently than
    do schools that have been less successful?

-   What vocational education practices and approaches appear most
    promising in promoting academic achievement and technical
    competence?

-   What is the relationship between vocational education improvement
    and school reforms underway in many states and local communities?

-   Does federal vocational education policy, as implemented by states
    and districts, facilitate or impede implementation of effective
    programs of study at the secondary level?

     What is the quality of vocational education and to what extent are
    federal strategies for improving vocational education quality
    reflected in actual classroom practice?

Perkins III continues to emphasize several basic reform strategies, the
genesis of which was in Perkins II; e.g., integrating academic and
vocational education, linking secondary and postsecondary vocational
programs, and broadening vocational curriculum beyond its traditional
emphasis on entry-level job preparation. It is therefore important to
examine whether nearly a decade of federal efforts to improve the
quality of programs have found their way into teaching approaches and
classroom organization. This examination will be guided by several
research questions, such as

-   Are states and communities making progress in implementing key
    reform strategies emphasized in Perkins III? What other approaches
    reflect quality in vocational education?

-   To what extent has professional development been used to support
    these strategies? What kind of preparation and qualifications do
    vocational teachers have that will help them make these
    improvements?

-   How does the quality of vocational instruction and teaching vary
    among different institutions (comprehensive high schools, vocational
    high schools, and area vocational schools), different communities,
    or the demographic characteristics of students? What factors affect
    the depth or quality of vocational education?

-   What are the barriers to improving the quality of vocational
    practice?

# 

# 

C. Effectiveness of Promising Vocational Education Strategies

In recent years, a variety of programs and reforms have been introduced
to improve the outcomes of secondary vocational students. Some
approaches have been narrowly conceived (making minor adjustments to a
small set of courses or activities) while others involve broader
curriculum and institutional changes. Many of these reform models and
practices have become popular, expanding to more schools or becoming
institutionalized in state policy. Some have been supported with federal
grants.

Until recently, however, few studies have been conducted of sufficient
size and rigor to provide evidence on how well and why these reforms
work. Schools have to a large extent relied on limited information that
such interventions as Tech-Prep, career academies, career clusters, and
High Schools that Work are effective in improving student performance.
Now that these and other reforms have been in place for some time, it is
reasonable for policymakers and educators to seek answers to the
following questions:

![](media/image8.wmf){width="4.95in" height="2.6in"}

 Do these strategies, when well implemented, improve student outcomes?

Although developers and advocates of particular reform models have long
cited student success, in fact there is little concrete evidence of the
programs' positive impacts. Studies found that implementation of these
initiatives has been uneven, which has in some cases stymied efforts to
conduct large-scale evaluations of effectiveness. With longer
experience, however, examples of outstanding implementation have emerged
that can be used as test cases. Of particular interest are comprehensive
program models that can be applied to either vocational students or as
part of whole-school reform. Research on these models should address the
following questions:

-   To what extent do these promising interventions make a difference in
    key student outcomes: academic achievement, technical competence,
    postsecondary enrollment, or employment?

-   What other outcomes are associated with the new strategies (e.g.,
    technical literacy, work readiness, employer satisfaction)?

-   Are there differences in impacts for different groups of students
    (defined, for example, by demographics, socioeconomic status, skill
    level, or occupational interest)?

     What specific elements do approaches that seem to work have in
    common?

While knowing which programs or reforms make a difference in student
outcomes is important, so is understanding the elements that make some
initiatives successful. For policy and technical assistance purposes, it
is critical to determine whether key outcomes result primarily from, for
example, innovative instructional techniques, changes in administrative
structures, or simply raising core requirements. Looking across all of
the studies may provide answers to the following key questions:

-   Which features of reform are essential to producing positive
    outcomes?

-   Do the importance of these features vary across school settings?

#### 

#### 

#### 

#### 

####  

# POSTSECONDARY OCCUPATIONAL EDUCATION {#postsecondary-occupational-education .maintopic}

Vocational education is both a high school and postsecondary program,
although Perkins III, as written, seems primarily focused on high
schools. Research has shown that there are significant differences
between students in high school and in college with respect to the
reasons they enroll in vocational courses as well as in overall
enrollment trends. Moreover, the primary institutions that deliver
vocational training---high schools and community colleges---are
fundamentally different. Nonetheless, Perkins III makes few distinctions
between high schools and colleges with respect to federal objectives,
program strategies, or accountability requirements.

Two issues are quite distinctive about postsecondary occupational
education and will receive substantial attention under the NAVE. Growing
evidence on participation and outcomes suggests that vocational
education at this level is beneficial. Postsecondary vocational
education also has direct implications for workforce development.

A. Participation and Outcomes

In contrast to vocational education at the secondary level, occupational
program enrollments at the postsecondary level continue to grow at a
rate in keeping with the sub-baccalaureate level more generally. In
addition, available evidence clearly establishes a shift toward
enrollment of older students. Such a shift in the demographics
potentially signals a diversity of roles if not a changing role for
occupational education at the postsecondary level---which may have
implications for federal policy.

![](media/image9.wmf){width="5.208333333333333in"
height="2.9027777777777777in"}

The economic benefits of postsecondary occupational education are a key
measure of the importance and impact of occupational programs. Most
analyses in the literature find a significant return to an associate's
degree. However, there are two caveats: first, there is evidence to
suggest that the benefits are uneven;[^9] and second, very few students
complete an associate's degree.[^10] NAVE will examine both the
participation and benefits of postsecondary occupational education. In
addition to the benefits for those who complete a degree, NAVE will
examine the benefits of occupational education by field, for those who
do not earn degrees, and for particular sub-groups of students.

 Who participates in postsecondary occupational education and why?

Knowing the characteristics of those who enroll, and how enrollments are
changing provides an important context for studying postsecondary
occupational education. Of growing importance is an older student
population that pursues more nontraditional pathways. The key questions
to be addressed are:

-   Who enrolls in postsecondary occupational education and how has this
    pattern changed? How do the characteristics of those who pursue
    occupational education compare with those in nonoccupational
    programs at the sub-baccalaureate level?

-   In what types of institutions do postsecondary occupational students
    enroll?

-   What are the occupational enrollments by program area? Have
    enrollments by program area changed over time? Are patterns of
    participation in nontraditional programs changing? How important is
    "not-for-credit" course taking?

-   Has there been a shift in the timing of when postsecondary
    occupational students enroll?

-   What are the educational goals of postsecondary occupational
    students? Does "reason for enrollment" vary by student
    characteristics?

     What is the contribution of occupational education at the
    postsecondary level?

The increasing prevalence of degree noncompletion suggests the need to
both understand the trends in and to more fully document the
contribution of postsecondary occupational education to wages and
earnings beyond simply looking at those who attain a degree.

-   How important is degree completion compared with skill acquisition
    (i.e., are alternative measures, such as industry-generated
    credentials, more beneficial in the labor market than traditional
    measures?) Are degrees less beneficial and thus less important in
    some fields?

-   What is the contribution of postsecondary occupational education to
    wages and earnings?

-   Does the impact vary by student characteristics, course taking, or
    field of study?

# 

# 

B. Postsecondary Occupational Education and Its Alignment with WIA

A major issue at the postsecondary level is coordinating occupational
programs with the workforce development system. When Congress enacted
Perkins III and the WIA, it believed that a plethora of job training
programs created excessive administrative burden upon states and
discouraged access to services. Therefore, Congress is likely to be
particularly interested in the relationship between Perkins and the WIA.

Workforce development is likely to be affected not only by the
institutional alignment of Perkins and WIA, but also by the extent to
which students complete their training. Although there are still many
occupations that require only minimal training and lower-level
education, occupations\
requiring at least an associate's degree are projected to grow faster
and with larger projected\
numerical increases than those requiring less education or
training.[^11] Despite (or perhaps as a consequence of) increased
enrollments in postsecondary education, degree completion rates are low
and are decreasing at the sub-baccalaureate level.[^12] Of particular
concern is the disproportionately high and growing rate of postsecondary
dropping out among "disadvantaged students" (such as students of lower
socioeconomic status, minority students, and those with lower high
school grades). Given the diversity of the students who attend
postsecondary institutions, the\
explanation for the comparatively low completion rates is unclear and
more than likely many-layered. The NAVE intends to shed light on this
issue by better understanding educational pathways and their
implications for workforce development.

![](media/image10.wmf){width="4.938888888888889in"
height="2.9833333333333334in"}

 What role does postsecondary occupational education play in
development of state and local workforce development strategies?

The 1998 WIA presents opportunities to integrate occupational education
with workforce investment systems (through the optional use of
state-unified plans of WIA and Perkins and the provision of workforce
development services, through one-stop career centers, at postsecondary\
providers). Several basic questions need to be addressed:

-   What is the relationship between local workforce boards and
    postsecondary occupational programs? To what extent do local
    workforce boards impact postsecondary occupational programs?

-   What is the interaction/relationship between postsecondary
    institutions and one-stops? Are one-stops used to integrate
    postsecondary occupational education and other workforce training
    programs?

     What is the quality of postsecondary occupational programs (as
    measured by available indicators)?

Several dimensions of program quality are the: (1) nature of programs
offered (as measured by faculty quality and responsiveness to labor
market demand); (2) "the program of study" students actually
pursue;[^13] and (3) results (as measured by employer satisfaction).
Relevant research questions include

-   What are the qualifications of occupational faculty (compared to
    past/academic 2-yr. faculty)? To what extent are postsecondary
    occupational faculty involved in professional development activities
    and of what type?

-   Are the occupational programs offered related to local labor market
    needs, and do they change adequately in response to local labor
    market needs (e.g., do enrollment trends coincide with anticipated
    labor market demand)?

-   What is the projected level of education needed for mid-level
    occupations, and do students attain the level of education
    consistent with projected labor market requirements? Do educational
    requirements vary by occupational field or by geographical region?

-   Are employers satisfied with postsecondary occupational education?
    To what extent are they involved in postsecondary occupational
    programs (e.g., prevalence of co-ops, involvement on advisory
    boards, use of postsecondary institutions in hiring practices etc.)?
    Are students satisfied with postsecondary occupational education?

-   Now that states are no longer required to set aside funds for a sex
    equity coordinator or special populations, how are these issues
    addressed?

-   What is the typical postsecondary occupational education pathway?
    How efficiently do occupational students progress toward goal
    completion (i.e., are course-taking patterns coherent and/or of a
    meaningful quantity or are students simply "milling around")? What
    are the persistence patterns of occupational students?

-   As measured by degree/certificate attainment or critical number of
    courses, are completion rates of occupational students commensurate
    with those of non-occupational students at the sub-baccalaureate
    level? Do completion rates vary by student characteristics (old
    versus young; by goal)?

-   What are the factors that contribute to longer persistence and
    higher completion rates? How is persistence and completion affected
    by differences in student goals and pathways?

# 

#  

# PROGRAM MANAGEMENT {#program-management .maintopic}

Federal vocational legislation has for some time used funding and
accountability provisions to achieve specific policy goals. Perkins III
is no exception. Perhaps the most significant changes in the new
legislation involve these program management tools. Understanding how
the new provisions are implemented, and the consequences of them, will
be an important issue for Congress and for the NAVE.

A. Accountability

Increased accountability is a major component of Perkins III. While
states have been expected to gather information on student outcomes
since 1990, few states have been able to consistently and systematically
do so. The 1998 Act raises the requirements for state reporting of
student outcome data, and the potential rewards and consequences for
states that can and cannot do so.

Perkins III requires each state to develop a system of measurements and
to establish expected levels of performance in four categories:

1.  Student attainment of academic, vocational, and technical skill
    proficiencies.

2.  Completion of a secondary or postsecondary degree or credential.

3.  Placement and retention in postsecondary education, advanced
    training, employment or the military.

4.  Participation and completion of programs that lead to nontraditional
    employment.

In addition, each state is required to report on the progress of special
populations with respect to each of these categories.

![](media/image11.wmf){width="5.209027777777778in"
height="2.823611111111111in"}

The major changes in Perkins III accountability have less to do with the
types of measures that are required than they do with how performance
data, once collected, is to be used. Perkins III requires states to
negotiate quantifiable levels of performance[^14] with the Department of
Education and to report yearly to the Secretary on progress made in
meeting these standards. Moreover, the Secretary is to make this
information available publicly and to compile state-by state
comparisons. Finally, Perkins III raises the potential stakes associated
with performance. Failure to meet state-level performance standards
could eventually result in loss of Perkins funds. As a reward, however,
section 503 of the WIA provides incentive grants to states that exceed
performance levels under the Perkins Act, Adult Education and Family
Literacy Act, and WIA Title I.

Given the significance of these provisions, NAVE intends to address two
main issues regarding accountability:

 Have states, school systems and colleges developed appropriate
measures, standards, and data-gathering systems?

Studies undertaken in the early 1990s of initial efforts to implement
Perkins II accountability requirements raised an important concern that
remains relevant today. States were generally able to meet (or exceed)
legislative requirements, but the quality of their systems was uneven. A
more recent article suggests that "states have made dramatic progress
toward meaningful accountability systems," perhaps spurred by the
realization that accountability is not a passing fad.[^15] Anecdotal
information indicates that Perkins III accountability requirements have
been taken much more seriously by federal and state officials than was
the case under the previous Act. However, several critical questions
need to be addressed:

-   Are data systems in place to produce reliable information about
    student performance?

-   Are local entities in a state using similar measures and assessing
    outcomes for the same population of students?

-   Have challenging, yet realistic, standards of performance been
    established?

-   Why have some states made greater progress in developing
    accountability systems for vocational education than others?

-   To what extent are states and local entities able to integrate WIA
    and Perkins performance accountability requirements?

     How are Perkins performance report results used to manage and
    improve programs?

In addition to highlighting data quality concerns, the earlier
accountability studies of Perkins III also questioned the utility of the
data collected. Local educators reported seeing little value in the
information they were asked to gather and report. They therefore failed
to use the data as intended for program improvement, providing even less
incentive for them to ensure high data quality. Given this history, it
will be important to examine several questions about the impact of the
new requirements:

-   Are states and communities using the data gathered to manage and
    improve their programs (i.e., are one-stop clients given access to
    this information)? Have accountability requirements resulted in any
    unintended effects?

-   Are the incentive payment provisions credible?

-   Can state reported performance data be aggregated to produce a
    national profile? Can it be used to compare state performance?

B. Funding Issues

It has been frequently observed that federal policy is a "blunt
instrument" for change. However, nowhere is the impact of federal
legislation more direct and immediate than the rules governing
allocation of funds. The Congress that enacted Perkins III was guided by
two major principles regarding funds (1) "drive" more money to local
grantees, and (2) provide greater flexibility regarding the use of
funds.

![](media/image12.wmf){width="4.940277777777778in"
height="2.765277777777778in"}

Toward these goals, the Act (1) eliminated a 10.5 percent set-aside for
single parents and displaced homemakers, enabling Congress to direct
that 85 percent (up from 75 percent) of funds should go to local
programs; (2) created a 10 percent reserve fund which may be distributed
outside of the intra-state formula; (3) eliminated the requirement for a
state sex equity coordinator while mandating that \$60--150,000 of state
leadership funds be used for sex equity services; (4) eliminated rules
regarding prioritizing local funds to serve programs with the highest
concentration of special populations; and (5) changed the secondary
school intra-state formula to simplify but not significantly alter
funding allocations. Also relevant was the elimination of an 8 percent
set-aside for school programs as Congress replaced the Job Training
Partnership Act with WIA. NAVE will examine both the allocation and use
of funds under the new Perkins provisions, including:

 To what extent have the new provisions affected how funds are
allocated?

-   How have changes in the intra-state formula, and reserve fund,
    altered the amount and distribution of funds at the local level?

-   Have states changed the allocation of funds between secondary and
    postsecondary education?

     How has the use of funds changed, if at all?

-   What has been the effect of eliminating set-asides and other
    provisions for special

-   populations?

-   How do states use "state leadership" funds?

-   What priorities are emphasized by local education agencies and
    postsecondary institutions in their use of Perkins funds? To what
    extent does the use of funds support priorities in Perkins III?

### 

### 

### 

###  

# OTHER MANDATED ISSUES {#other-mandated-issues .maintopic}

Congress mandated an assessment covering a broad range of issues. The
preceding discussion addressed a general approach to nearly all of the
topics in the congressional mandate---as well as some others. Two topics
not addressed, however, are (1) teacher supply and demand, and (2) the
role of technology and distance learning in vocational education. The
current plan for examining these other key issues include:

Teacher Supply and Demand

NAVE plans to utilize findings from a study of the supply and demand for
vocational education teachers that is currently being conducted by
NCRVE. Complementing this research will be information gathered from the
comparative case studies (described previously).

Technology and Distance Learning

Several efforts are currently underway to track the growth of distance
learning programs of study, and to understand the instructional role of
technology. NAVE plans to summarize this research as it relates to
occupational training. How vocational curriculum responds to changes in
commercial technology is a topic to be explored in the case studies.

# APPENDIX A {#appendix-a .maintopic}

![](media/image1.png){width="0.9493055555555555in" height="0.78125in"}

> **[National Assessment of Vocational Education]{.smallcaps}**
>
> **Independent Advisory Council**

  ------------------------------- ---------------------------------------
                                  

  **Naomi Nightingale**\          **Stephen Hamilton**\
  *(Panel Chair)\                 Cornell University
  LA Metropolitan Transit         
  Authority*                      

  **Paul Cole**\                  **James Jacobs**\
  *(Panel Co-Vice Chair)\         Macomb Community College
  New York State, AFL-CIO*        

  **Russ McCampbell**\            **Jack Jennings**\
  *(Panel Co-Vice Chair) Missouri Center on Education Policy
  State\                          
  Director* *of Vocational        
  Education*                      

  **Karl Anderson**\              **Dale Kalkofen**\
  Saturn Corporation              *Associate Superintendent for School
                                  Reform*\
                                  Memphis City Schools

  **June Atkinson**\              **Chris King**\
  *North Carolina State Director\ LBJ School\
  of Vocational Education*        University of Texas at Austin

  **John Bishop**\                **JoAnn Kister**\
  Cornell University              *Ohio State Director of Vocational and
                                  Adult Education*

  **Gene Bottoms**\               **Mark Milliron**\
  Southern Regional Education     *Oracle Corporation*
  Board                           

  **Betsy Brand**\                **Katherine Oliver**\
  American Youth Policy Forum     *Maryland State Superintendent of
                                  Career Technology\
                                  and Adult Learning*

  **Jay Cummings**\               **Robert Runkle**\
  *Assistant to Texas             *Director*, Berks County Career and
  Commissioner of\                Technology Center
  Education*                      

  **Philip Day, Jr.**\            **Tony Sarmiento**\
  *Chancellor\                    *AFL-CIO*
  City College of San Francisco*  

  **James Folkening**\            **Ellen O'Brien Saunders**\
  Postsecondary Services\         *Director*, Workforce Training and
  Michigan Department of          Education Board\
  Education                       Washington State
  ------------------------------- ---------------------------------------

[^1]: 1982, 1989, and 1994.

[^2]: Figure 2 shows the trends in federal funding for vocational
    education over the past 20 years, and compares funding for
    vocational education with overall funding for Department of
    Education programs.

[^3]: Figure 3 compares ED funding of high schools from three major
    sources---the Perkins Act, Title I of Improving America's Schools
    Act (IASA), and the TRIO program.

[^4]: Ferran Mane, *Trends in the Payoff to Academic and
    Occupational-Specific Skills: The Short and Medium Run Returns to
    Academic and Vocational High School Courses for Non-College Bound
    Students* (Working paper \# 98-07) (Ithaca, NY: Center for Advanced
    Human Resource Studies, Cornell University).

[^5]: National Assessment of Vocational Education, *Final Report to
    Congress,* Volume I, July 1994.

[^6]: Ibid.

[^7]: Ibid.

[^8]: Alan Hershey, Marsha Silverberg, Joshua Haimson, Paula Hudis, and
    Russell Jackson, *Expanding Options for Students: Report to Congress
    on the National Evaluation of School-to-Work Implementation*
    (Princeton, NJ: Mathematica Policy Research, October 1998).

[^9]: Not all fields of study lead to significant positive rates of
    return. See W. Norton Grubb, *Learning and Earning in the Middle:
    The Economic Benefits of Sub-baccalaureate Education* (New York:
    Teachers College).

[^10]: Based on the HS&B sophomore cohort, of all students who enroll in
    a 2-year college, about 16 percent attain an associate's degree. In
    contrast, nearly 60 percent of those who enroll in a 4-year college
    complete at least a bachelor's degree. Thomas J. Kane and Cecilia
    Elena Rouse, "The Community College: Educating Students at the
    Margin Between College and Work," *Journal of Economic Perspectives*
    (Winter 1999): 63--84.

[^11]: George T. Silvestri, "Occupational Employment Projections to
    2006," *Monthly Labor Review* (November 1997):\
    58--83.

[^12]: D. Boesel and E. Fredland, *Is There Too Much Emphasis on Getting
    a College Degree?* (Washington, DC: National Library of Education,
    June 1998).

[^13]: That is, what courses are students taking or combining, and what
    is their progress?

[^14]: States can renegotiate levels of performance prior to the third
    program year covered by the state plan. Timelines for the NAVE
    report will allow it to describe any changes in standards that may
    occur, but not the resulting outcomes.

[^15]: Mikala Rahn and Patricia Holmes, "Accountability Systems:
    Performance Standards and Assessment," *NCRVE Centerpoint* (March
    1999).