**Edging Toward Effectiveness:\
Examining Postsecondary Occupational Education**

**W. Norton Grubb**[^1]

**University of California, Berkeley**

**May 6--7, 1999**

**Independent Advisory Panel Meeting**

**National Assessment of Vocational Education**

In many ways postsecondary occupational education (PSOE) is a
stepchild---even a stepchild of a stepchild. The institutions where it
takes place---community colleges, technical institutes, some area
vocational schools, other public training centers, private proprietary
schools---are not well known and are often of low status. In federal
policy, these institutions are often afterthoughts: they do not benefit
from the large programs (Chapter 1, aid for bilingual and handicapped
students) aimed at K--12 schooling; they are not the principal targets
of the large amounts of federal funds in postsecondary student aid; and
federal aid for vocational education, one of the few federal programs
providing funding to both secondary and postsecondary programs, has
always been written with secondary education in mind. And within the
institution where most PSOE occurs---community colleges---occupational
education often suffers again from second-class status relative to
academic or transfer education.

But PSOE is more important than its status suggests. It is now the locus
of most vocational education, as least vocational education understood
as preparation for immediate employment below the baccalaureate
level,[^2] since vocational education in high schools has either been
transformed into more general programs ("education through occupations"
or "college and careers") or has dissipated into random course
offerings. The majority of students in community colleges---about 60
percent[^3]---are in occupational programs; almost all the rest are
enrolled for broadly occupational purposes, even if they think these
will be realized through academic programs and transfer. PSOE fulfills
the promise of providing postsecondary education---increasing viewed as
necessary for both individual mobility and employer needs---for the
large numbers of occupations for which baccalaureate degrees are not
necessary, and indeed the economic returns to sub-baccalaureate
education are highest for those who complete occupational programs and
find related employment (Grubb 1999). The promises of "lifelong
learning," so often empty rhetorical flourishes, are probably best
realized within PSOE, with its large number of older students returning
for skill upgrading or retraining. And when comprehensive institutions
like community colleges provide PSOE, their "bridging function"---their
ability to create bridges or paths from one purpose to
another---involves PSOE in a web of other programs and purposes. In this
role, students who enter community colleges for remedial purposes,
"experimenters" looking for a life path, welfare recipients, dislocated
workers, or others in job training and adult education programs can find
a way into the mainstream of economic life.[^4]

If PSOE is more important than is often recognized, then it merits more
research than is currently available. Like PSOE itself, research on this
corner of American education has been badly neglected. Most of the
research on vocational education concentrates on the secondary level;
most of the research on community colleges focuses on transfer or
academic education, or on the high-status "economic development" role of
providing training for particular firms. And so PSOE has rarely had its
day in the sun.[^5] I therefore begin (rather than end) this paper with
the researcher's conventional plea: *more research is needed*.

In this paper I suggest a variety of topics for the National Assessment
of Vocational Education to support as it reviews progress under the Carl
Perkins Vocational and Applied Technology Education Amendments of 1998,
or Perkins III. The task most closely related to the legislation itself
is to examine how community colleges and other postsecondary
institutions have responded to the key requirements of Perkins III, and
I cover these in Section I. They include the development of
accountability measures; the incorporation of academic competencies into
PSOE and the integration of academic and occupational education;
tech-prep efforts to link high schools and community colleges; the
allocation of federal funds; and the types of institutions providing
PSOE.

However, while federal legislation for occupational education has
addressed several important ways of improving PSOE, it has never been
especially comprehensive, particularly in its approach to postsecondary
issues. In Section II I therefore shift to a variety of topics intended
to address the effectiveness of PSOE, using a five-part conception of
the institutional conditions necessary for effectiveness as well as
proposing certain research related to outcomes. This section is
consistent with the broader purpose underlying federal legislation for
vocational education, which has always stressed program improvement.

In Section III I examine four issues that are perhaps more important
than any of the others presented in this paper: the question of who
counts as "completers" in PSOE, almost certainly the most vexing
question to PSOE administrators; the "new fluidity" in postsecondary
education and its potential threat to established programs; the question
of why federal legislation in many areas related to vocational education
and training has often had unsystematic effects; and the correlative
question of what role the federal government should perform. These
issues are not raised directly in Perkins III, and these issues are not
included in the congressional mandate to NAVE. But they represent the
most pressing current developments in PSOE and the central dilemmas in
federal policy toward PSOE. Therefore ignoring them may consign NAVE to
irrelevance and the eighth circle of hell,[^6] and can only ensure that
federal legislation remains out of touch with current problems.

In Section IV I draw together the various *issues* presented in the
first three sections into *research strategies*. Typically, each kind of
research encompasses several issues; for example, the case studies of
community colleges I propose could encompass a variety of issues,
including local practices about integration, accountability, and tech
prep as well as participation with other education and training
providers. Furthermore, the various research strategies are related to
one another: a broad sample of providers could respond to questionnaire
surveys; a smaller sample of these institutions could become case
studies; and a smaller sample of these could be the starting point for
"community case studies" examining the variety of programs in a
community. I will therefore cross-reference the research presented in
Section IV to the issues presented in Sections I--III, and vice versa.

Finally, since I propose too many issues for NAVE to support, I briefly
present my own priorities for research in Section V.

Too often, proposals for research become little more than laundry lists
of topics with little coherence, and certainly this paper could be
critiqued for precisely that flaw. However, there is one central issue
underlying virtually all of the research proposed in this paper*: how do
we make postsecondary occupational education more effective?* This
question is addressed in a few ways in Perkins III (Section I), though
the approaches to effectiveness could be broadened, as I argue in
Section II. And failure to consider some basic issues of current
conditions will mean that future legislation is not especially powerful
in improving effectiveness, which is why the topics in Section III are
so important.

The stress on effectiveness is also, in my view, the way to think about
equity in PSOE. The current equity problems are not, with a few
exceptions, those of access to PSOE, since community colleges and other
institutions providing PSOE are usually open-access institutions with
low tuition, low opportunity costs, and high levels of both state and
federal subsidies. The problems experienced by the groups who are the
usual focus of equity efforts---low-income students, minorities, women
seeking access to nontraditional employment, older students, dislocated
workers, handicapped students, and other "nontraditional" students---are
instead those of movement through these programs, of completion rather
than noncompletion, of the coherence of learning and its value in the
labor market. While there are some issues specific to equity and
particular groups, particularly those considering support services (in
Section II.3), the greatest improvement in equity would come simply from
improving the quality of PSOE for all students.

Finally, the issue of effectiveness is one way of creating a compelling
vision, or policy narrative,[^7] for postsecondary occupational
education. In secondary vocational education, such a compelling vision
has emerged in the "new vocationalism" (or "education through
occupations," as I prefer to label it, or David Stern's conception of
"college and careers")---in which academic and occupational competencies
are integrated, in which school-based and work-based learning are
linked, in which high schools are thoroughly restructured into smaller
units (academies, clusters, or pathways), and in which broader forms of
occupationally oriented education lead to further education as well as
employment. But a comparable vision has not yet emerged for
postsecondary occupational education, aside from the vision underlying
most federal postsecondary policy---which has been simply to expand
access through student aid and, more recently, tax credits. A compelling
complement (not alternative) to the vision of access is a vision of
effectiveness---and this idea underlies much of the research proposed in
this paper.

I. The Issues in Current Federal Legislation

The current legislation contains several specific provisions that are
particularly germane to PSOE. They can be investigated in various ways,
including questionnaires to postsecondary institutions (the basis for
the previous NAVE). However, because questionnaire results often seem
unreliable, I recommend that they be supplemented in virtually every
issue with case studies, to understand better *how* the intentions of
Perkins III are being met (or ignored).

I.1: Accountability and Performance Measures

Many observers feel that the most novel element of Perkins III is its
requirement that states develop at least four measures of state
performance. States that do not make adequate progress in their own
performance measures are threatened with having their state plans
disapproved and their funding eliminated (Section 121e) and, in
addition, incentive funding under the Workforce Investment Act (WIA) is
allocated on the basis of performance measures for Perkins III, job
training, and adult education. The final consequence of the state
performance measures adopted is that local programs are to be evaluated
by the "State adjusted levels of performance," and those not making
substantial progress are required to develop local improvement plans
(Section 123). Unlike the former legislation, therefore, there might be
real consequences to poor performance under these new performance
measures. I note that the development of these accountability systems
requires both *state* measures, used for comparison across states, and
*local* measures, used to identify low-performing local programs and
improve their performance so state measures can improve. While local
indicators should be tied in some way to state measures, they could be
different, and the sanctions a state uses with local program could go
beyond the improvement plan required in Perkins III.

The new provision generates a variety of obvious questions. The first is
what state measures the different states have adopted, and then what
local indicators they have adopted---questions that can be resolved by
examining state plans (see IV.1). However, if performance measures are
to be used to improve the quality of local programs, a more interesting
issue is what states have done first to require or encourage local
programs to meet local standards, including what forms of technical
assistance or additional financing they have provided in addition to
threats like potential reductions in funding. This question can be
addressed both from the vantage of the state, through questionnaires or
interviews with state directors (see IV.2), and from the vantage of the
local providers, through questionnaire studies (IV.3). Finally, the most
important issue is how these performance measures and standards have
influenced local programs, where it's important to recognize that these
influences might be unintended and negative---like the "creaming" of
students and the cooking of data sometimes seen in JTPA programs---as
well as intended and positive. While questionnaires to local programs
may elicit some of these changes (see IV.3), corroboration from local
case studies (IV.4) would be necessary, particularly in uncovering some
of the less positive potential effects of accountability.

I note that most of the four core measures required by Perkins III
involve issues raised in other research proposed in this paper. The
first core measure, related to academic and technical competencies,
could be reflected by passage of licenses or industry-generated
standards, for example (see III.1). The second measure, on completing
postsecondary degrees or credentials, involves the complex issue of who
"completers" are (see III.1), and some states may have used measures of
completion other than certificates or associate's degrees. (Many
colleges have begun to define early-exit options, partly for welfare
recipients, that recognize earlier stages of "completion.") The third
core measure, related to placement and retention in employment, could
most readily be addressed with the kinds of Unemployment Insurance (UI)
data reviewed briefly in Section II.6. Therefore, information about
accountability measures being adopted may shed some light on other PSOE
issues.

Some local programs, particularly technical institutes and community
colleges, are also likely to be subject to performance measures required
by WIA, either for adult services or for adult education. A question
related more to the relationship among federal programs (see II.5) is
how Perkins performance measures interact with WIA measures, or whether
there are inconsistencies that make compliance with various indicators
difficult. Again, this issue can be investigated to some extent through
local questionnaires (IV.3), though case studies of local programs
(IV.4) and community case studies examining the interaction among
programs (IV.5) would be valuable in corroborating the responses of
local institutions.

One question raised with the new performance requirements is whether the
costs of complying with Perkins III outweigh the benefits to
PSOE---particularly in those states that have made little progress since
Perkins II in developing data systems that can be used for
accountability. (This is a new version of an old complaint about federal
policy: "the tail wags the dog," in the sense that small amounts of
federal funding have disproportionate influences on local programs.)
Federal funding accounts for a small proportion of PSOE and of most
local programs, especially in comprehensive institutions like community
colleges (see I.4). Some institutions may decide that the burdens of
participating in performance measures outweigh the benefits of small
sums; therefore research should be alert for indications that local
programs have forgone federal resources. This issue is in turn related
to the question of what role the federal government should play in PSOE
(III.4): if the federal government hands out trivial sums of money and
requires inordinate costs for compliance, then its help on balance may
be no help at all.

I.2: Incorporating Academic Competencies and Integrating Academic and
Occupational Education

Perkins III is filled with language exhorting and requiring PSOE to
incorporate a range of academic, occupational, and technical
competencies, and the first of eight requirements of local uses of funds
specifically provides for the "integration of academics with vocational
and technical education through a coherent sequence of courses." It must
be acknowledged that the purpose of stimulating broader and more
integrated programs, through this and other provisions in Section 135
(including the "all aspects of an industry" provision), are accompanied
by language permitting local programs to spend federal funds on a wide
variety of activities, so that local programs can comply with federal
requirements without paying the least amount of attention to this
requirement for improvement (see III.3). Nonetheless, one of the central
changes of Perkins II, continued in Perkins III, was the attempt to move
away from the kind of "narrow vocationalism" often critiqued by the
business community (e.g., Committee for Economic Development 1985, p.
15) toward a broader and more integrated conception of PSOE, parallel to
a similar effort at the secondary level.

The most obvious task is to learn more about what local institutions
have done to implement this provision. The prior NAVE conducted surveys
of local programs to ascertain changing practices, and such questions
could easily be included in questionnaires to local institutions (see
IV.3). In my view, the earlier study generated local responses that were
exaggerated; for example, 38.5 percent of institutions reported starting
tandem courses---coordinated academic and occupational courses---before
1991--92, even though all my other experience with community college
indicates that such courses are quite rare.[^8] (See also Badway 1998,
for a more extended argument and evidence that the prior NAVE had a low
threshold for compliance with the integration requirements in Perkins
II.) One possible interpretation is that, since the previous surveys
were done quite soon after Perkins II was enacted, they represented
hopes and dreams rather than reforms already implemented. Now, however,
with an additional six years for implementation, it should be easier to
see what local programs have put in place.

However, the possibility for exaggeration is still high, particularly
because these surveys are completed by central administrators whose
knowledge of what happens in classrooms is usually somewhere between
limited and zero (Grubb and Associates 1999, Ch. 8). Therefore, any
replication of this survey should be complemented by case studies of
local colleges and other providers (see IV.4) to see precisely how
academic competencies are incorporated or integrated into PSOE. These
local case studies should ideally include interviews with occupational
deans and instructors to get their perspectives on such integration, on
what has been accomplished, and on the (inevitable) barriers to
integration. However, while self-assessments of teaching practices are
reasonably accurate under some circumstances, they are invariably
exaggerations (Mayer 1999) and they fail to convey precisely what
happens within classrooms---as the observation-based writing of Grubb
and Associates (1999, Ch. 7) and Perrin (1998) illustrates. Therefore,
the case studies should include observations of occupational classes and
integrated classes to examine the extent and form that integration of
academic competencies take.

Some community colleges collect good longitudinal evidence on their
students, and potentially this could be used to examine the effects of
integration on progress and subsequent success (see IV.11). For example,
where some occupational classes are linked with academic classes and
some are not, it may be possible to compare the subsequent success of
students in the two different settings. While obvious problems of
selection and self-selection may be present, NAVE researchers ought at
least be alert to opportunities for further analyses of local data.

I.3: Tech Prep

Starting in Perkins II, tech prep is a vision of linking the last two
years of high school with two years of community college, where the
links could (and should) include the reform and integration of curricula
at the two levels as well as mechanisms to enhance the transition
between the two. At least in theory, tech prep is a mechanism that would
encourage more "noncollege bound" students to attend postsecondary
education, and to enroll without the need for remedial education that
has become so pervasive.

However, like many of the aspects of work force development programs,
the term "tech prep" has taken on several meanings among local
educators: sometimes it refers to efforts to integrate academic and
vocational education, sometimes to simple articulation agreements
between high schools and colleges, and sometimes to school-to-work
programs incorporating some work-based learning. There's been a
substantial amount of research already on tech-prep programs, including
a national evaluation (Hershey Silverberg, Haimson, Hudis, and Jackson
1998) as well as many studies by NCRVE (e.g., Bragg, Layton, and Hammons
1994; Bragg 1996; Grubb, Badway, Bell, and Kraskouskas 1996); Debra
Bragg is now in the midst of an evaluation of the effects of tech prep,
supported by NCRVE. Most of these find incomplete implementation of tech
prep, with local programs usually incorporating components (like
articulation agreements or integrated curricula at the high school
level) but very few developing truly integrated programs.

Another finding, particularly important for postsecondary institutions,
is that the most substantial changes have been at the high school level,
with changes (including curricular changes) in community colleges
relatively less common. In the early days of tech prep this was due to
the fact that students ostensibly in tech-prep programs had not yet
reached community colleges; even in Bragg's current research, the number
of completers is still small. Three research strategies seem obvious.
One is to continue Bragg's research, now scheduled to end this year, as
a way of gathering more information about the eventual completion (or
noncompletion) of students in the postsecondary component of tech-prep
programs. Another is to include examination of tech-prep programs in the
institutional case studies (IV.4) to learn more about the curriculum
developments (if any) and the barriers at the postsecondary level; the
marginal cost of such research would be small. The final strategy is to
incorporate some questions into the questionnaire surveys of community
colleges (IV.3), parallel to those asked in the previous NAVE; as is
true for the integration of academic and occupational education, the
earlier results constitute a benchmark and it will now be possible to
see how much progress there has been and what kinds of postsecondary
institutions seem to have made the most progress.

However, it's difficult (at least for me) to imagine that further
research on tech prep will lead to powerful new insights. The finding of
incomplete implementation of tech prep has emerged over and over again,
and is likely to be rediscovered yet one more time. Therefore, I suspect
that the most interesting question to be asked of tech prep programs is
why federal legislation has so often led to partial reforms and
"business as usual," even when the underlying ideas are widely
celebrated (see IV.3).

I.4: Where Does the Money Go?

A basic question to be addressed in NAVE is the question of how federal
funds under Perkins III are spent. This can be addressed in several
ways: by examining state plans and supporting fiscal documents (IV.1),
and by examining both the *pattern* of spending and the *process* by
which decisions about Perkins funds are made in local institutions,
through case studies (IV.4).

In the past, a great deal of Perkins funds have been spent on normal
expenditures and routine reforms---new equipment, for example, or
updating courses in ways that might be thought to be the responsibility
of local programs using state and local resources. This is explicitly
allowed in Perkins III, which permits funds to be spent to "initiate,
improve, expand, and modernize quality vocational and technical
education programs," but it is contrary to the intent of program
improvement in any more fundamental sense. One potential reason why this
has been true is that most local public institutions are funded through
formulas that seem to have academic programs in mind. There is no
recognition for the variation in costs among programs, and no routine
allocation for capital expenditures; thus high-cost occupational
programs, and those with extensive requirements for up-to-date equipment
and workshops, must often grovel for extra money and must use federal
funds for what seem like routine expenditures of keeping programs up to
date. One way to examine whether this common complaint explains the uses
of federal funds is to examine whether the patterns of spending, and the
process by which federal funds are allocated, are different in states
with cost differentials in funding and explicit funding for capital
equipment, or whether the patterns and processes are different in
institutions like technical institutes with funding dedicated to
occupational programs, compared to patterns in comprehensive community
colleges. This is also part of examining why federal funds for
occupational education and other work force development programs have so
often been ineffective in fostering improvement (see IV.3).

Another dimension of the flow of federal funds is how large they are
relative to other funds spent in local institutions. In both
questionnaires (IV.3) and case studies (IV.4), the relative importance
of Perkins funds can be ascertained. Then one question is whether
institutions with relatively high fractions of their funds from Perkins
use the revenues in ways different from those with relatively low levels
of funding.

This research could take the opportunity to track all federal funds into
a variety of postsecondary institutions. This can be done both through
case studies (IV.4) and questionnaires (IV.3) completed by fiscal
officers, at least to determine the variety of federal funds from
student aid as well as other sources (like the Fund for the Improvement
of Postsecondary Education or the National Science Foundation). Grants
and loans should also be examined from the federal perspective, using
federal data (e.g., from the National Postsecondary Student Aid Surveys)
on the amounts of grants and loans to community colleges and technical
institutes (see IV.8). One puzzle that has never been fully answered is
why community colleges and technical institutes receive such relatively
small fractions of grants and loans, compared with the numbers of
students (or low-income students) they serve (e.g., Grubb and Tuma
1991), and a specific investigation of this issue might generate
recommendations of how to change the administration of grants and
loans---at either the federal or local levels---to facilitate the flow
of these funds to PSOE students.

Finally, an enormous new source of federal funds was "appropriated" in
1996 with the Hope Tax Credits and the Lifelong Learning Tax Credits,
amounting to perhaps \$6 billion per year. The amounts provided through
the Hope Tax Credits were pegged to average community college tuition
levels, as if these credits could enable all students to attend at least
two years of postsecondary education free of charge.[^9] However, tax
credits are awkward and potentially inequitable mechanisms of funding,
particularly to low-income students whose taxes may not be sufficient to
allow tax credits to be of any use. Therefore a separate study should
try to estimate the flow of these two tax credits to students in
sub-baccalaureate postsecondary institutions generally and in PSOE in
particular. Alternative methodologies may be necessary: one is to survey
students to see what proportion of those in community colleges and
technical institutes (and in four-year colleges) applied for these tax
credits; from the federal perspective, IRS records can provide
information on the total amounts of these tax credits claimed and they
can in theory identify the types of institutions attended since
taxpayers qualifying for these credits must identify the institutions
they attended. The distribution of these tax credits is a sufficiently
broad issue that NAVE might be able to join forces with other research
organizations---for example, the College Board, with its continuing
interest in the funding of postsecondary education---to examine these
funding patterns.

Federal funding for PSOE is particularly complex because the funds are
partly direct expenditures to institutions (e.g., Perkins III and
FIPSE), partly direct expenditures to students (grants and loans), and
partly tax expenditures to students (in Hope and Lifelong Learning Tax
Credits). One question that should be incorporated in any overall
examination of the federal role in PSOE (see IV.4) is whether this mix
of funding---inadvertent and unplanned---is the most effective and
equitable way to support PSOE.

I.5: Types of Postsecondary Institutions

While the majority of PSOE, and of funds through Perkins III, is surely
spent in community colleges, there are many other postsecondary
institutions that provide PSOE, many of these poorly
understood---except, perhaps, by those within them. These include
technical colleges and institutes, which provide two-year associate's
degrees but are (unlike comprehensive community colleges) generally
confined to occupational subjects;[^10] area vocational schools (the
Regional Occupation Programs and Centers in California, the BOCES in New
York), which often provide various certificates to adults; postsecondary
programs operated by local educational authorities, which include adult
schools in many states and some vocational schools and career centers;
some vocational centers operated primarily with job training funds that
may also receive some Perkins funds and provide certificates similar to
those offered in community colleges or area vocational schools; and
other hybrids and consortia too varied to name. Perkins III recognizes
institutions of higher education (community colleges and technical
institutes), LEAs providing postsecondary education, area vocational
schools, postsecondary institutions operated by the Bureau of Indian
Affairs to any Indian tribe, educational service agencies, and consortia
(Sec. 3 (10)).

A useful exercise would be to catalogue the variety of these
postsecondary institutions, to determine the amounts of federal funds
going to each, the enrollments and types of credentials granted, and the
trends in these enrollments. The sources of data could include national
data sets like NPSAS and the Beginning Postsecondary Surveys, state
enrollment and expenditures data, and federal sources on the allocation
of Perkins funds. There's a boundary problem to confront: the types of
institutions considered will differ if the starting point is (a) the
universe of institutions receiving Perkins funds; (b) the institutions
reported in a particular national data set, like NPSAS or BPS; or (c)
data collected from states on their definitions of postsecondary
programs. One task, therefore, would be to examine how various ways of
identifying these institutions affects the numbers and types of
institutions considered.

To examine trends, however, data over time would be necessary. It
appears, for example, that technical institutes are converting to
comprehensive community colleges,[^11] as part of a larger historical
process of replacing specialized institutions with comprehensive ones.
Area vocational schools also seem relatively unimportant, except perhaps
in certain states that have maintained systems of them; in other cases
they have been upgraded into technical institutes. But the information
on which my observations are based are random and incomplete; a more
systemic study could identify the patterns throughout the country more
accurately.

While the descriptive purposes of examining enrollments and funding
patterns are useful, a purpose more consistent with the theme of
effectiveness is to examine the types of programs offered. There are
certainly some highly sophisticated programs being offered by these
varied institutions, like the two-year programs in technical fields
offered by the Texas State Technical Colleges. On the other hand, many
area vocational schools and career centers appear to offer short (e.g,
15-week) programs for relatively unskilled occupations, quite similar to
the short-term job training programs offered by JTPA that have proved so
ineffective.[^12] More generally, the economic benefits of short PSOE
programs are varied and uncertain (Grubb 1999), at least compared to
longer programs and associate's degrees. Any study of the varied types
of postsecondary institutions should try to uncover data on their
effectiveness---either by finding local follow-up studies or perhaps
state UI wage record data (see II.6 below)---to see whether more can be
learned about the labor market value of such programs. In the absence of
more specific state or local data, however, my concern is that
institutions providing short-term training are relatively ineffective.

I.6: Substate Allocation of Perkins Funds

One persistent aggravation among PSOE administrators is that federal
legislation requires states to allocate Perkins funds according to the
number of students receiving Pell grants (Section 132), presumably in an
effort to allocate more funding to institutions with low-income
students. However, the number of Pell recipients is not necessarily a
good indicator either of the number of students in PSOE or of the
concentration of low-income students, partly because Pell grant
recipients include many academic students and therefore don't reflect
needs for PSOE; partly because many PSOE students are part-time and
therefore ineligible for Pell grants; partly because many of them are
older and do not appear to apply for grants and loans; and partly
because the rates of receiving Pell grants may be related to
factors---e.g., the amount of information provided to students, the
extent of assistance with student aid, the sophistication of students in
dealing with institutions---that have nothing to do with the needs for
PSOE. One study might therefore examine alternative methods of
allocating funds, to see what difference it makes to the overall
distribution and to weigh the consequences of different allocation
formulas. Alternatives might include, for example, the number of
students in PSOE, where enrollments in PSOE could be defined in various
ways; the total enrollments in local institutions, on the assumption
that Perkins aid should be potentially available to all students in
sub-baccalaureate institutions; the number of PSOE students weighted by
local income, to allocate more resources to institutions in low-income
areas; the number of PSOE students weighted by the local unemployment
rate, to distribute more funds to low employment areas; or some measure
of the size of the sub-baccalaureate labor market, to tie funding to the
extent of appropriate employment rather than enrollments. Each of these
(and other) measures would change the distribution of funds and the
incentives on local programs. This information would also be useful in
considering the potential direction that federal policy toward PSOE
might take (see IV.4).

II\. Beyond Perkins III: Five Principles for Effective PSOE Programs

Since the early 1960s, one of the central purposes of federal aid for
vocational education has been program improvement. Most recently, in
Perkins II and now in Perkins III, improvement has principally been
defined as the integration of academic and occupational education, the
creation of tech-prep programs, and the development of performance
measures to reflect outcomes. However, there are many more dimensions to
effectiveness than these three particular approaches, and NAVE ought to
consider research related to effectiveness and program improvement that
goes beyond the conception embedded in Perkins III.

In examining effectiveness, there is often no substitute for analyzing
outcomes, and I propose several ways to research outcomes in Section
II.6. However, often it is impossible to measure outcomes. In other
cases, changes need to be considered before long-run outcome measures
can be available, and in any event evidence of poor outcomes is
literally useless without some sense of what kinds of improvements might
enhance outcomes. Therefore it is almost always necessary to have some
idea of what elements of PSOE programs are necessary for greater
effectiveness, in addition to (and sometimes instead of) information
about outcomes. Over the last several years I have developed five
criteria to judge the effectiveness of many types of occupational
education and training programs, based on my reading of the available
evaluation and econometric literature and on the characteristics of
particularly effective programs.[^13] Others may differ about whether
these are precisely the right criteria and about how important each of
them is, but they at least provide a starting point for thinking about
effectiveness---and the potential research that might illuminate
improved practice. Each of these five criteria therefore serves as a
starting point for at least two kinds of research: determining what
conventional or modal practices in PSOE are, and identifying promising
practices for others to emulate.

II.1: Labor Market Information and Connections

*Effective programs understand the local labor market, and target those
jobs that are likely to employ individuals with community college
credentials, with relatively high earnings, strong employment growth,
and opportunities for individual advancement.* Some community colleges
do this through careful local needs assessments; others, particularly
those with extensive co-operative programs, accomplish this by
maintaining stable links with local employers. Programs in high
employment areas, and many programs during the current economic boom,
can get away with ignoring this recommendation, of course, though
probably at the expense of long-run effectiveness. But those colleges
that fail to consider the hiring requirements and the quality of jobs
for which they provide occupational preparation are likely to place
individuals in minimum-wage positions with few prospects for
advancement, with dismal results over the long run.

There are various mechanisms by which local institutions can keep
abreast of labor market patterns and trends, including formal labor
market assessments, advisory committees, the local contacts of
individual instructors, placement office, follow-up studies, and---most
impressive of all---the contacts afforded by high-quality co-operative
education programs. However, the extent of these efforts varies
substantially, and some---for example, advisory committees that meet
once a year for ceremonial functions, or placement offices that stress
low-quality "stay-in-school" jobs, or instructor contacts that are
uneven and idiosyncratic---are not much use at all (Grubb 1996b, Ch. 6).
Thus, contact with local labor market conditions is far from automatic
in PSOE, and may need improvement in many cases.

Two kinds of research are possible. One is to ascertain how the
mechanisms that potentially connect local programs with employers and
labor markets actually function, through local case studies (see IV.4).
An adjunct to such a study would try to identify best practices from
around the country, to provide guidance to community colleges and other
PSOE providers about the most effective ways of maintaining connections
with local employers.

The second and complementary study would question *employers* about
their practices---particularly about their proclivity to hire from local
PSOE programs rather than other sources, their preferences for
experience over formal schooling, the kinds of promotion practices they
use and the role of formal PSOE in promotion. Some observers have
concluded that many employers---at least those at the sub-professional
level---hire on the basis of skills attained regardless of the source of
those skills, so that community colleges students compete with those
trained in the military, those who have learned through experience, and
others who have picked up skills in informal or self-taught ways. In
other cases, licensing requirements, company policy, and formal or
informal agreements with local colleges create tighter linkages between
formal PSOE and employment. Further analysis of employer
practices---through either survey methods, interviews (see IV.8), or the
analysis of several local labor markets (see IV.5)---would help clarify
where employment practices encourage rather than discourage hiring from
local PSOE programs.

II.2: Curriculum and Pedagogy

*Effective programs contain an appropriate mix of academic (or remedial,
or basic) education, occupational skills, and work-based learning, in
the best cases integrated with one another. In addition, while flexible
and innovative scheduling may be necessary, the intensity of both
academic and vocational education must be appropriate to the jobs being
targeted. Finally, effective programs pay attention to the pedagogy of
everything they teach, whether classroom-based or work-based.*

This principle, which really embodies three interrelated conditions for
effectiveness, clarifies the obvious point that PSOE is an *educational*
program. If the fundamental conditions for learning are not present,
then PSOE can at best serve as a signal of motivation, ability, or some
other valued characteristic but cannot make students more productive.
Research under this principle is potentially vast, since there is
virtually no analysis in this country of the effectiveness of different
approaches to curriculum and instruction (Achtenhagen and Grubb 1999).
One strand includes the research on the incorporation of academic
competencies proposed above in Section I.2.

Another strand could include more focused research on work-based
learning, and its integration with other components of instruction.
Despite the recent interest in this country in work-based learning, and
the small amounts of funding from the School-to-Work Opportunities Act,
work-based learning is comparatively rare in most community college
(except in certain fields like nursing). The questionnaire-based surveys
of community colleges provide one opportunity to ask about the extent of
work-based learning, though such results would no doubt simply replicate
those available from earlier studies (e.g., the Bragg studies cited
earlier). The case studies (see IV.4) are probably more effective ways
to ascertain the role of work-based learning, including the resources
available to support it, the nature of its integration (if any) with the
rest of the curriculum, and (potentially) measures of effectiveness if
the case study institutions have adequate data on student outcomes.

The other topic that has been virtually unexplored in the area of
curriculum and pedagogy is the nature of teaching in PSOE. Aside from
one chapter describing occupational classes in Richardson, Fisk, and
Okun (1983) and an analysis of occupational teaching in several chapters
of my recent book (Grubb and Associates 1999), there's been nothing
written about the nature and challenges of teaching in postsecondary
occupational subjects. (For a broader review of the international
literature, see Achtenhagen and Grubb 1999.) From the work that has been
done, I have argued that occupational teaching is in many ways more
complex than academic teaching: it incorporates a greater variety of
competencies or "intelligences"; more settings, typically including
classrooms, workshops, and sometimes job placements; different and often
more demanding forms of reading, writing, and mathematics than are
taught in conventional academic classes; a greater number of literacies
including specialized diagrams and charts as well as conventional
writing and mathematics; and distinctly different approaches to teaching
that I have labeled "skills" versus "systems" approaches, paralleling
the distinctions between behaviorist and constructivist teaching in
other subjects. Yet for all this complexity, PSOE instructors receive
virtually no preparation in teaching (save in Iowa and a few isolated
community colleges), and most postsecondary instructors learn how to
teach through trial and error. The classroom observations I have
recommended as part of institutional case studies (see IV.4) could
readily be extended into a special study of its own. This special study
might, for example, concentrate on PSOE teaching within selected
occupational areas, or within occupational areas subject to licensing or
to industry-generated standards (see IV.1). or within institutions where
instructor training is practiced. In any of these forms, its central
purpose would be to further investigate the nature of pedagogy in PSOE,
the origin of different instructors' approaches to teaching, and the
potential ways of improving instruction.

Finally, I should acknowledge the difficulty of addressing questions of
the appropriate intensity for PSOE programs. The dismal outcomes of job
training programs are due in substantial part to their limited duration
and intensity; within PSOE, the economic returns are clearly higher to
those completing associate's degrees than to individuals with
certificates taking shorter amounts of time or those with varying
amounts of coursework. If more is better, then how much is enough? And
when is more too much? One way to examine this question is to examine
the industry-generated standards and licensing requirements described in
Section IV.1, since these specify from the *employer's* perspective how
much education is enough for various positions. Another is to examine
the debates within occupational fields about the intensity of
postsecondary education, though this discussion can take place only in
occupations where there is some kind of organization---either an
industry association, a licensing body, or an association of
instructors---to convene such a discussion. For the moment, the former
seems an easier starting point.

II.3: Support Services

*Effective programs provide a variety of support services, as
appropriate given the needs of their students.* While PSOE may be
principally an educational program, an institution that fails to
consider the need for support services may have low completion rates and
fail for that reason alone. In community colleges and other forms of
PSOE, five kinds of support services seem particularly important for
students in occupational education:

-   **Guidance and counseling mechanisms:** Particularly for
    "experimenters" unsure of what they want to do, or dislocated
    workers looking for alternative occupation, or welfare recipients
    seeking a way back into the labor force, guidance and counseling is
    particularly important. Guidance and counseling *mechanisms* could
    include introductory courses or project-based efforts to get
    students to learn about occupational alternatives and the efforts of
    occupational instructors themselves, and not simply the efforts of
    guidance counselors in their offices. The Puente program in
    California, which creates learning communities of instructors along
    with counselors, provides another interesting model of integrating
    counseling into instructional programs.

    However, as in high schools, there are many complaints about the
    adequacy and type of counseling available in community colleges and
    other postsecondary institutions; counselors are often criticized
    for providing information solely about academic issues and transfer,
    neglecting career-oriented counseling. While instructors and
    administrators can describe what happens within their own
    institution, there seems to be no central source of information
    about postsecondary guidance and counseling, no established
    conception of the developmental processes involved in adult
    decisions about careers, no conventions about good practice. NAVE
    could provide a substantial service to PSOE by carrying out, perhaps
    as part of case studies (IV.4), an analysis of what's now available
    in postsecondary guidance and counseling and some method of
    developing conceptions of good practice.

-   **Remedial/developmental education:** The problem of underprepared
    students in all postsecondary programs has grown substantially, and
    PSOE is no exception. Community colleges and technical institutes
    typically provide some forms of remedial or developmental education,
    though the type and quality of these programs varies enormously, and
    almost nothing is known about the effectiveness of these programs in
    helping students complete their goals. In the best cases, individual
    instructors have developed their own approaches (often
    constructivist) through trial and error, or a developmental studies
    department has devised a coherent policy, or instructors have linked
    developmental courses with occupational courses in learning
    communities. (See also I.2 above, on integrating academic and
    occupational education for remedial purposes). In others forms of
    PSOE, remedial education may be missing, or students may be directed
    to adult education programs of varying (and mostly abysmal)
    quality.[^14]

    The nature and effectiveness of remedial/developmental education is
    a large subject, applicable to most postsecondary institutions, and
    this is not an issue that NAVE can readily research on its own. As
    with many other questions, the case studies of local institutions
    (see IV.4) provide one opportunity to examine the nature of these
    efforts, their relationships (if any) with occupational programs,
    and (if institutions have good enough data) their effectiveness.
    However, NAVE might also explore the possibilities for joint
    research with other government agencies and centers concerned with
    the quality of postsecondary education.

-   **Retention programs:** The issue of who is a completer, and whether
    completion as defined by certificates and associate's degrees is
    important, is one of the most vexing questions in PSOE, one that I
    examine more fully in Section IV.1. For the moment, however, it is
    sufficient to point out that community colleges have made various
    efforts to increase retention and completion. They can be
    categorized as sorting, supporting, connecting, and transforming
    strategies (Beatty-Gunther 1994), and the best community colleges
    use a variety of them. Again, institutional case studies (see IV.4)
    provide the easiest ways of learning more about the nature of these
    efforts and their interactions with occupational programs more
    specifically.

-   **Placement efforts:** In occupation-specific forms of education,
    the benefits may be lost unless an individual finds employment
    related to his or her field of study. Results from national data
    confirm that the economic returns to PSOE are much higher when
    individuals find related employment, especially among women (Grubb
    1997 or 1999); and some preliminary California results based on UI
    data corroborate these findings. Therefore, occupational education
    by itself may be necessary but not sufficient to gain access to
    well-paid employment, and placement efforts may be necessary as
    well. Again, local case studies provide low-cost ways of examining
    what community colleges and other local PSOE programs now offer.

-   **Services for students with disabilities:** Because community
    colleges are open-access institutions with relatively low spending
    levels, the diagnostic procedures to identify students with learning
    disabilities are often weak, and services to help such students are
    often inadequate. Community college instructors report that they
    suspect having students with learning disabilities, but do not know
    how to help them and have nowhere to refer them. Such students often
    end up in remedial classes, which may not be appropriate. While I am
    not the right person to recommend how to address this issue, a
    fuller analysis of support services would consider carefully the
    availability of such disabilities and their effects on the progress
    of the students themselves as well as their effects on other
    students.

II.4: Connections to Other Programs

*Effective programs provide their clients or students with pathways or
"ladders" of further education opportunities, so they can continue their
education and training when they are able to.* One issue of creating
such pathways in PSOE involves the link between high school and
community college, in the tech-prep programs described in Section I.3
above. The other common linkage is that between community colleges and
four-year colleges, in articulation mechanisms to smooth the process of
transfer. In many occupational areas---e.g., business, computers and
information technology, engineering technology, some health
occupations---the possibilities of transfer are substantial, and both
instructors and institutions provide encouragement and support (like
transfer centers); in other areas---automotive areas, metal trades,
construction trades, cosmetology---there are no comparable
baccalaureate-level programs and so articulation mechanisms are scarce.
The possibility of and support for transfer should be one of the issues
incorporated into local case studies (see IV.4).

However, the most important kinds of connections for NAVE to investigate
are probably the connections among different types of occupational
education and training programs. These connections---the efforts to
create more coherent systems of work force development encompassing
short-term job training, adult education, welfare-to-work programs,
training for incumbent workers, and occupational education in
postsecondary institutions---have been the subject of substantial
research by NCRVE (see Grubb et al. 1999, for the latest effort) as well
as research by the earlier NAVE (e.g., Hollinger and Harvey 1994).
However, changes in federal legislation---particularly the Workforce
Investment Act (WIA) of 1998 and the changes in welfare legislation
passed in 1996---have substantially changed the conditions within which
local programs operate, and therefore new research may be warranted. In
particular, the emphasis of both welfare legislation and WIA on "work
first" conceptions---in which individuals are first placed in employment
without worrying about their competencies---contradicts the human
capital emphasis of PSOE, and the resolution of this conflict is a
central issue.

The issues involved in relations with other programs include the
provision of services and special programs for welfare recipients (if
any); the ways PSOE institutions participate in state and local
Workforce Investment Boards; the incorporation of PSOE into information
and services provided by local one-stop centers, which play a central
role under WIA; the eligibility of PSOE for the Individual Training
Accounts (ITAs) provided some individuals, the numbers and types of
individuals who actually use ITAs to attend community colleges and other
forms of PSOE; and the difficulty PSOE institutions have in meeting WIA
accountability measures (see I.1 above). The period within which NAVE
will operate---with data being collected roughly between summer of 1999
and summer of 2001---will be one of substantial turmoil, as states
create Workforce Investment Boards under WIA and begin planning local
efforts, and therefore it may be possible only to understand how PSOE is
participating in this process and not what the outcomes are.[^15]

One way to carry out this research is again to incorporate questions
about participation in WIA into the questionnaire surveys (see IV.3) and
local case studies (IV.4). However, because these issues involve
interactions between PSOE and other programs, it would be far better to
conduct research examining WIA efforts and welfare programs as well as
PSOE. This can be accomplished with community case studies, where
several communities are identified and all the programs within these
communities providing PSOE, job training under WIA and welfare, adult
education, and other forms of work force development are examined to
learn about their interactions (see IV.5).[^16] In this way the
perspectives of each program about the others can be ascertained, rather
than viewing the cooperation among programs solely from the vantage of
community colleges and other PSOE institutions. Of course, the
communities selected for such studies could be a subset of the
communities in which the institutional case studies (see IV.3) are
conducted, so that the community cases studies are effectively
extensions of the institutional case studies.

II.5: Data and Improvement

*Effective programs collect appropriate information about their results
and use these to improve their quality.* In the case of PSOE, these
informational issues include the development of accountability measures,
examined above in Section I.1, and the development of state and local
data on employment outcomes, examined in the next section.

II.6: Outcome Measures: National and State Sources

To many researchers and policymakers in particular, outcome
measures---rather than the elements of program structure I have covered
in Sections II.1 to II.5---are the only reliable indicators of
effectiveness. The good news is that better data have become available
in the last ten years, both from national data sets and from state data
sets based on UI wage record data.

In terms of national studies, I have recently surveyed the available
literature for the Community College Research Center (Grubb 1999), and I
see no reason for replicating that survey. There are, however, some
topics that have not been extensively studied, and if NAVE wants to
invest in further analyses of national data, it should concentrate on
these issues. One of them is the problem (see also IV.1) of whether
students who complete small amounts of PSOE benefit or not; this
requires disentangling the effects of field of study, the relatedness of
education and subsequent employment, and student intentions (i.e., for
upgrade training) on economic returns, since it's likely that all of
these influence the value of small amounts of education. A second issue
that merits more exploration is the effect of finding education-related
employment, since this has been investigated in only one study (Grubb
1997); there are obvious selection and self-selection effects to
consider, as well as alternative ways to define education and employment
as being related or not. Third, the effect of local economic conditions
and of the business cycle on the employment benefits of
sub-baccalaureate education have not been studied, though they are
arguably quite important. A study reanalyzing certain data
sets---particularly the Survey of Income and Program Participation and
the National Longitudinal Study of Youth---could address these
unanswered questions (see IV.8).

States have made great progress in developing data based on UI data. A
few states have been leaders in these efforts, including California,
Washington, Texas, Florida, and North Carolina. However, the analyses
they have carried out so far are relatively unsophisticated (partly for
political reasons), and have generally been confined to a few
cross-tabulations. One valuable NAVE project would therefore be to take
one or more of these state data sets and carry out more sophisticated
multivariate analyses (see IV.10). In addition to carrying out such
analyses, such a study should consider how to translate complex findings
into simpler terms, since policymakers and administrators are unable to
read regression coefficients. In addition, I propose in Section IV.1
using these state data, in conjunction with other information, to carry
out further analysis of the complex issue of completion.

One other use of state UI data would be to conduct an outlier study. In
the school effectiveness literature, particularly effective (and
ineffective) schools were identified with regression analysis of test
scores controlling for family background and school resources; schools
with high positive residuals were then considered especially effective,
and case studies were conducted to see how they differed from other
schools. Similarly, in the realm of job training, Chris King and his
colleagues (King, Lawson, Olson, Trott, and Baj 1998) selected
especially effective JTPA programs in Texas and Illinois based on
earnings over 155 percent of poverty and on continuous employment. The
outliers they visited were committed to a structured process for
determining high demand and emerging occupations; they emphasized
occupational skills training rather than short-term job search
assistance and on-the-job training; they stressed certain support
services, including intake, assessment, counseling, and case management
continuing throughout the program and afterwards; and they constrained
the choices of training activities, referring individuals only to
approved providers and only in high-demand or emerging occupations.
(These characteristics correspond to the principles articulated in
Sections II.1, II.2, and II.3 above.) A similar outlier study could be
conducted of community colleges, for example, by using regression
analyses of a state's UI data, identifying programs with high placement
rates and earnings controlling for student characteristics and local
economic conditions, and visiting those programs to see what
characteristics of effectiveness are present. Such a study would build
on the reanalysis of a state's data (see IV.10), and then add a small
number of additional case studies.

III\. Beyond Perkins III: Other Crucial Issues

The topics suggested in Section II move beyond the provisions of the
Perkins Amendments of 1998, but they do not address other fundamental
issues about the context of PSOE and about the nature of federal
legislation. In discussions with occupational educators and researchers,
a small number of these other issues emerge again and again, and they
could form the basis for other research studies.

III.1: The Meaning of Completion

By far the most vexing problem to occupational educators is the question
of what constitutes completion. In conventional analyses, completion of
PSOE is marked by receipt of an associate's degree---typically requiring
about 60 credits including a sequence of occupational courses, related
academic courses, and general education requirements---or a certificate,
typically requiring about 30 credits in occupational subjects only
(though some certificates are shorter or longer).[^17] Associate's
degrees and certificates are typically defined and recognized by states,
they show up on transcripts, and in my reading of course catalogues they
usually require a relatively coherent group of courses. Completion has
substantial economic benefits, including "program" or "sheepskin"
effects (Grubb 1999); many policymakers feel it should be encouraged,
and completion rates are among the performance measures required by both
Perkins III and by WIA. But rates of completion in PSOE are quite low,
and this has been a continual embarrassment to community colleges and
other providers.

However, many students who leave PSOE without associate's degrees or
certificates may nevertheless complete what they set out to accomplish.
These include at least two distinct categories:

\(1\) Some students, particularly those needing upgrade training, intend
to enroll only for a course or two. These students are more likely to be
employed, to be older, and to continue in the same job. Students needing
retraining, including dislocated workers, may switch occupations, but
they too may enroll for only the number of courses they need to find a
new position; they are also likely to be experienced workers. One way to
identify these "completers" is through their intentions; some community
colleges that collect data on student intentions report that large
fractions of older students claim to want only a few courses. Under some
circumstances, the same may be true of students needing remedial
education.

\(2\) Students may complete credentials other than associate's degrees
or certificates. These include licenses, including those required for
health professions, cosmetologists and other personal service workers,
some construction trades, aviation mechanics licensed by the Federal
Aviation Administration; the kinds of industry-generated certificates
(IGCs) created by industry associations with voluntary standards,
including such groups as NATEF (the National Automotive Technicians
Educational Foundation) and the American Welding Society; and some IGCs
created by individual companies, such as the credentials offered by
Novell and Microsoft to those individuals training specifically on their
software. If students learn enough to pass such licenses and IGCs, they
may leave PSOE and appear to be drop-outs when they have in reality
completed a credential with, arguably, some value in the labor market.
Finally, some community colleges have begun to modularize occupational
programs and create shorter credentials or early-exit options, partly to
recognize the reality that students leave before completing credentials,
and partly to provide welfare recipients with credentials with the
allowable training period. Whether these credentials have any market
value is unclear, though no doubt there is local lore in specific
colleges.

The fraction of "dropouts" who are really "completers" in one of these
two senses is currently unknown. PSOE administrators and instructors can
relate many stories about specific students who are completers in these
ways, but their implication---that *all* PSOE students are really
completers---is surely not true. Given the importance of this question
to conceptions of effectiveness, as well as to information about the
changing conditions of PSOE, NAVE might carry out studies specifically
to examine the conception of completion.

There are at least two parts to such research. One, of course, is the
investigation of student intentions. The ideal study would either create
a survey of student intentions---or use one that community colleges are
now using---and then verify the results through student interviews (see
IV.7). The importance of interviews is that some students who are really
"experimenters" may exaggerate their intentions, or may state intentions
(like transfer and completion of a B.A.) that are socially acceptable
but unstable;[^18] it's equally possible that older students searching
for a substantial change in employment state intentions for completing
only a few courses until they figure out the alternatives. This
component of research could also try to link student intentions to
information about student outcomes, either from institutional records or
from UI data, to examine the match among intentions, courses and degrees
completed, and employment outcomes.

The second component requires examining the receipt of licenses, IGCs,
and other non-standard credentials. The problem is that the sources of
information are varied. Of course, questionnaires to students about such
credentials is one source of information, though response rates to
student surveys are typically very low. Information about state-required
licenses could come from states, and in a few cases from colleges that
administer licensing exams; information about IGCs could come from the
industry associations themselves. In theory these different sources of
information could generate information about which individual students
in a specific college and in a particular cohort have completed such
credentials, and then UI data or other follow-up data could be used to
ascertain their subsequent employment and earnings. One alternative is
to choose a few community colleges in relatively well-defined labor
markets, in states with adequate UI data, and use them as test cases to
develop both the methodology of track alternative credentials and their
effects on employment.[^19]

The development of IGCs presents still another issue for NAVE to
address. In this country, interest in skill standards has led to a
series of pilot projects funded by the Department of Labor to create
standards in 22 industries. However, IGCs are evidence of
industry-generated standards (or IGSs) that have already been created by
industry associations who feel a need for such standards and
credentials, rather than being artificially created by government
sponsorship. A study of such IGSs and IGCs could focus on a limited
number of such industry groups, ascertain how widespread such standards
and credentials are, how widely they have been adopted by community
colleges and other PSOE providers, what their use by employers is, and
what value they have in hiring, promotion, wage setting, and other
aspects of employment.[^20] The appearance and the apparent expansion of
IGSs and IGCs provides a natural experiment in skill standards, one that
could shed considerable light not only on the developing conditions of
PSOE but also on the conditions under which skill standards are likely
to be effective.

III.2: The Environment of PSOE: The "New Fluidity"

There's a new day dawning in PSOE, many observers report. Students
accumulate credits from many different institutions, including four-year
and two-year colleges; they may learn skills from company-sponsored
courses or IGCs, or though the military or experience, or on the
Internet or through distance learning so that the spatial limitations of
conventional providers are irrelevant. Employers care about skills, not
credentials, so the accumulation of skills from different sources,
without credentials attached to them, is still effective in providing
access to employment. In this fluid environment, the notion of entering
a "two-year" college, attending full time, progressing steadily through
the requirements for general education and a major, and receiving an
associate's degree in roughly two years is a rare event, an image of the
past (Schneider and Stevenson 1999, Ch. 9). In the "new fluidity"---to
give it a name---students draw on many providers of education and
training, and the educational programs so carefully put into place by
community college instructors debating the coherence of their offerings
are all but irrelevant. The "new fluidity"---which may not be so new,
after all---is sometimes invoked to explain why there are so few
completers in community colleges, and sometimes used to describe a world
of fierce competition among providers, where community colleges may lose
their "customers" to the Internet, distance learning, other public and
private providers.

However, it isn't clear how realistic this portrait is, or how extensive
the mix-and-match practices of the "new fluidity" are.[^21] It isn't
clear how students signal their skills to employers, for example, or how
the pattern of earning credits from many providers can teach the
competencies like SCANS skills and other higher order competencies that
some employers say they want. Sometimes this scenario seems like
alarmism, particularly of educators fearful of losing students, or
concerned that their carefully planned programs are being trashed by
student practices, and the reality might not match their fears. One
option, therefore, is to explore the "new fluidity" empirically,
particularly through community case studies (see IV.5 below) which would
examine the variety of education and training providers in several
communities---including distance and Internet providers---to see the
extent to which students move among them. In addition, it would be
critical to interview a sample of students (see IV.7), to see how they
are putting together their own programs. This would, of course, be
complementary to interviews with students intended to determine their
goals in enrolling in community colleges, and whether from the outset
their intentions include gathering credits from numerous sources.

If the "new fluidity" proves to be an accurate description of the
direction of PSOE, it raises questions about the appropriate role of
publicly funded institutions. It's possible that some kinds of education
are best left to private providers or to distance learning and the
Internet---for example, information-dense subjects for independent
students. PSOE may have a different niche---for example, subjects where
the integration of workshop and classroom learning is important, or
where constructivist or systems approaches to teaching higher order
competencies are important. But before it's possible to say what the
role of PSOE could be in the "new fluidity," it's important to
understand better what precisely is taking place.

III.3: The Limited Success of Federal Legislation

Federal legislation related to vocational education, school-to-work
programs, job training, adult education, and other aspects of work force
development has had, in my view, only limited success. On the one hand,
it has provided support for some good ideas---for example, the upgrading
of vocational education, the integration of vocational and academic
learning, the incorporation of work-based or service learning with
school-based learning, the efforts to link secondary and postsecondary
education, the development of yet other second-chance programs for the
long-term unemployed and welfare recipients. Virtually every innovation
in PSOE is funded in some way or other with federal funds, and the
impact of federal funding often seems much larger than the amount---"the
tail wags the dog," in the conventional expression. On the other hand,
the spread of these practices has been slow, partial, and incomplete. A
great deal of federal funding for occupational education is spent for
routine forms of program improvement---for example, upgrading equipment
and courses, which is something that occupational programs ought to be
doing routinely---and for relatively standard forms of remediation.
Perkins II fostered a great deal of curriculum integration, but largely
in the form of applied academics courses---the most pallid form of
integrating academic and vocational education. Tech prep and
school-to-work funds have been used to support bits and pieces of good
practice---some articulation agreements, some additional guidance and
counseling, some applied academics courses---but without doing much to
create coherent systems with widespread participation (Hershey,
Silverberg, Owens, and Hulsey 1998; Hershey et al. 1998).

Before examining the final question of what the federal government
should do in PSOE, in the next section, it's important to raise the
issue of why federal legislation in this arena has been so checkered in
its effects---or, more precisely, to see whether there's some empirical
research that might shed some light on this issue. There are several
hypotheses and fruitful avenues for research:

-   Many occupational educators report that state and local funding is
    simply inadequate for their programs, since there is (in most
    states) no recognition of the higher costs of certain occupational
    courses, of the special equipment and materials costs, and of the
    ancillary costs of establishing connections with employers including
    work-based placements. Therefore they have to scrounge for every
    additional dollar they can find, and federal funds are used in these
    ways too. The implication is that in states that do have funding for
    the higher costs of occupational programs, and routinized funding
    for capital equipment, it might be possible that federal funds are
    used in different ways.

-   The amount of federal funding in most PSOE institutions is very
    small---a dated estimate is that perhaps one to three percent of
    overall funding in community colleges, or two to four percent of
    funding for occupational education come from federal sources (Grubb
    and Stern 1989). With such small amounts of funding, programs cannot
    commit themselves to larger changes that require state and local
    funding---and so federal funds are used for "bits and pieces" of
    programs, not coherent system changes. One implication is that
    programs that receive relatively larger amounts of federal funding
    may use them in distinctly different ways.

-   Outside funds are often viewed differently than internal funds,
    since they may disappear, or have separate regulations attached to
    them that require administrators to keep them separate, or have
    participation and allocation requirements different from other
    funds. This usually means that outside funds are reserved for
    relatively peripheral activities rather than being integrated into
    mainstream programs.

These hypotheses, and others that could be generated through discussions
with federal and local administrators, could be investigated by
examining spending patterns and through interviews with local
administrators. Some of these questions could be integrated into
questionnaires or interviews with state directors of PSOE (see IV.2) and
into questionnaire studies and case studies (IV.3 and IV.4) of local
providers. Alternatively, it might be possible to identify some PSOE
institutions that have used federal funds in distinctly different ways,
and then identify what conditions of these institutions have influenced
this difference.

This question is, of course, much larger than federal legislation for
occupational education and related programs. Many federal programs---and
indeed most state programs---intended to enhance the effectiveness of
social programs including education have been faulted for low
effectiveness in instigating reform. However, before deciding what
future directions for policy might be, it is crucial to establish when
federal legislation has been effective and when its influence has been
only marginal.

III.4: What Should Federal PSOE Policy Be?

The culminating question, the one that in many ways motivates all the
research that NAVE carries out, is what PSOE policy might be, and
whether it should take a new direction in legislation subsequent to
Perkins III. The question here is not what this direction might be, but
whether there are research activities that might help determine the
options for federal policy.

There are at least five activities that might be helpful in addressing
this question:

-   A review of the principles of fiscal federalism, which specifies
    when a federal government might provide goods and services and when
    a state or local government is more appropriate.[^22] Some
    understanding of the strengths and weakness of state policy in PSOE,
    from IV.3 above, might be useful in this exercise.

-   A review of the principles and results of federal legislation in
    other areas of education and job training, including the
    effectiveness of such policies. For example, policy in K--12
    education has tended to provide funding through categorical programs
    like Title I rather than through student funding as in postsecondary
    policy.

-   An analysis of other countries and their efforts to create PSOE
    policies might be useful, particularly if the examination
    concentrated on countries reasonably like our own---for example,
    Australia with its federalist system, Great Britain with its
    free-market approaches---rather than countries like Germany and
    Japan with very different institutional and government histories.

-   A consensus-generating exercise might be useful, bringing together a
    variety of federal, state, and local administrators and
    policymakers, to identify the current strengths and weaknesses of
    different levels of government. This exercise might start, for
    example, with a paper outlining a broad array of options for federal
    policy---or for a joint conception of federal and state
    policy---based on other activities, and then have participants weigh
    the alternatives from a variety of perspectives.

-   All NAVE contractors could be required to include a section on
    implications for federal policy at the end of their reports. This
    will at least force individuals to question what potential
    implications there might be of the very different kinds of
    information they collect.

In general, these activities seem pallid in comparison with the task of
specifying new directions for federal policy. This is perhaps because
the broad outlines of policy are not usually created from rational
exercises like this, but rather from a more extended process of
discussion, debate, and consensus.[^23] However, because there has been
so little explicit thought given to what federal policy in the arena of
PSOE should be, NAVE could at least begin the discussion through the
activities I have outlined.

IV\. Research Activities

The issues I have raised so far do not always constitute well-defined
research activities; rather, they are questions that need to be answered
through various kinds of information. In this section I briefly outline
the kinds of research implied by the three previous sections, the issues
each kind of research could address, and some of the major problems in
designing the research. The detailed designs for research must await
another time and place.

In considering the research that NAVE can undertake, its timetable must
be carefully considered. If NAVE is to report to Congress by July 2002,
and requires the first half of 2002 to evaluate all its findings, and if
researchers therefore need to complete their reports by the end of 2001
and require the fall of that year for writing, then *at best* data
collection can take place between summer of 1999 and summer of 2001. But
interim state plans were due in April 1999, and many states will not
have definitive plans until April 2000; the school year 2000--2001 will
be, in effect, the first school year under Perkins III, though it will
be the last year of data collection for NAVE. Therefore research on
changes that Perkins III has introduced---particularly those having to
do with accountability---must rely on plans and perceptions rather than
steps already undertaken. Similarly, the participation of PSOE
institutions with WIA and adult education must rely on reports of early
planning stages, since WIA need not be implemented until July 1, 2000.
Fortunately, this timing problem is largely confined to accountability
and WIA since in other important issues---the other issues related to
Perkins III in Section I, the larger issues of effectiveness in Section
II, and the other topics raised in Section III---are relatively enduring
ones, where the questions have been relatively stable all during the
1990s.

IV.1: Examining State Plans

State plans are usually compliance-oriented documents, often containing
relatively little useful information. However, they do at least describe
the state performance measures and the any local indicators developed by
the states, and there will be useful in examining patterns in
accountability (see I.1).

IV.2: Surveying or Interviewing State Directors of PSOE

Several issues can be investigated through interviews or questionnaires
directed at state directors of postsecondary education, including the
nature of performance measures (I.1), the decisions on the allocation of
state funds (I.4), the nature of postsecondary institutions in the
states (I.5), the strengths and weaknesses of state departments with
responsibility for PSOE (useful in III.3), and the normative questions
of what federal policy should be (III.4). While forced-choice
questionnaires are cheaper, they do not permit exploration of
interesting topics. A hybrid approach would therefore be to administer
questionnaires in all states, and then to choose a sample of state
directors for interview. This sample could include, for example, the
states corresponding to local case studies (in IV.4) and community case
studies (IV.5); a sample of large states and a sample randomly chosen;
or a purposive sample of states known to have strong state policies with
another of states with weak policies.

IV.3: Questionnaire Surveys of Local Institutions

Many of the issues covered above require information from local
community colleges and other PSOE institutions. Indeed, almost all the
topics in Section I and II require information from local institutions,
to ask them how they use federal funds (I.2, I.3, I.4) and what
practices they now follow related to effectiveness (II.1 through II.4).
In addition, the critical question of what a completer is (see III.1)
can be partially answered, by asking local institutions what information
they collect about student intentions and alternative credentials, and
what that information reveals. Because this information will be
inconsistent among institutions, however, it will be impossible to
aggregate such results to the national level; instead such information
will be useful primarily for determining the quality of current data and
the range of responses.

Questionnaire surveys are the most efficient ways of getting such
information. The questionnaires developed for the prior NAVE can be used
as the basis for updated questions, in order to examine trends over
time. However, because local responses are likely to be exaggerated, my
recommendation is that no questionnaires be administered without case
studies to check on their validity, as I propose in the next section.

IV.4: Case Studies of Local Institutions

Local case studies are more reliable because they can interview many
different respondents---local PSOE administrators, PSOE instructors,
administrators and faculty from the academic side (to check, for
example, on the participation in curriculum integration), local interest
groups like senates and unions, and students (but see IV.7). They can
also examine practices cited; for example, if a college claims to be
integrating academic and occupational education in a particular way, or
reshaping their curricula for tech-prep students, investigators can
visit the classes and instructors to confirm these reforms. The issues
to be included in local case studies include all those incorporated into
the questionnaires mentioned in IV.3, including virtually all of the
issues in Sections I and II. In addition, a case study should include
questions about completers and noncompleters, relying on local data (see
III.1), and on local perceptions of the "new fluidity" (III.2). Finally,
local perceptions of state policy, a necessary component of examining
responses to federal legislation (III.3), could be included in these
case studies.

One way to carry out case studies is to identify a small set of
community colleges and PSOE providers and carry out such studies after
responses from questionnaires have been completed, so that the validity
of questionnaire responses can be checked. It's difficult to know how
many such case studies are necessary, since they need to reflect the
variety of urban, suburban, and rural institutions, with some other
institutional variants in states that have them; perhaps 20 community
colleges and 10 other types of PSOE institutions would be sufficient to
generate real results.

In my view, a central component of local case studies should include
classroom observations. Only through observation is it possible to
determine the extent of integration of different subjects including
remedial education (see I.2), for example, or the approaches to teaching
of occupational faculty (see II.2). If such observations can be
videotaped, then they could begin the creation of a permanent data base
for better understanding teaching in community colleges and PSOE
specifically. (See Stasz 1999, for a similar argument at the secondary
level.)

IV.5: Community Case Studies

Community case studies, examining all the education and training
providers in a community, could be carried out for two purposes: to
examine the linkages among programs (see II.4), particularly as programs
begin planning for WIA; and to describe the "new fluidity" (see III.2).
Communities for such case studies should be selected from the
communities where local institutional case studies (IV.4) are carried
out, so as not to duplicate the information collected from certain PSOE
institutions. The communities selected need to vary in certain
dimensions---particularly by income or unemployment rates and the
composition of local employment---and such communities need to be
relatively self-contained, so that it is possible to set limits on the
providers who are defined as operating within the community. In prior
NCRVE research, two communities in each of four states, or eight
communities in all, provided a wealth of information (McDonnell and
Grubb 1991; Grubb and McDonnell 1991 and 1995), and such a design might
be replicated.

IV.6: Questionnaire Studies of Faculty

The responses of PSOE faculty on questionnaires would be principally
useful in identifying the methods of integrating academic and
occupational education, and how widespread such practices are (see I.2);
any changes in response to tech prep (I.3); the role of employers and
other outside groups in setting curricula, including the role of
industry-generated credentials and standards (II.2 and III.1); the
approaches of instructors to pedagogy, including the question of whether
local institutions play any role in helping instructors with their
teaching (II.2); the interaction of instructors with support services,
particularly developmental education and guidance and counseling (II.3);
any influences on instructors of federally required performance measures
(I.1) or other locally generated accountability standards (II.4); and
faculty perceptions of the issue of completion and the "new fluidity"
(III.1 and III.2). In addition, there are at least two other studies of
postsecondary faculty---one supported by NCRVE (Brewer and Gray 1997)
and one by the Carnnegie Foundation for the Advancement of Teaching
(Huber 1998), and these ought to be consulted in designing any new
faculty surveys.

IV.7: Questionnaire Studies and Interviews of Students

Questionnaires to students would be useful in ascertaining their
perceptions of the value of many practices, including integration (I.2),
tech prep (I.3), and student support services like guidance and
counseling and remediation (II.3). However, the most critical focus for
student services should be the issue of student intentions, raised in
the issue of who counts as a "completer" (see III.1); and student
activity in attending multiple institutions, as part of examining the
"new fluidity" (III.2). Unfortunately, for these purposes questionnaires
may not be sufficient, since what counts is not responses to
forced-choice questions---which are likely to exaggerate---but rather
the stability and precision of these plans. Therefore some student
interviews should be used to check on the reliability of student
questionnaire responses. Of particular value in such questionnaires
would be deriving their school and work histories---that is, the
different periods of work and schooling they pursue after high school,
trying to determine how related and coherent---rather than fluid and
unplanned---such pathways are.[^24]

Locating samples of community college students---especially samples of
central interest like noncompleters or leavers---can be quite difficult,
particularly since the records community colleges maintain may not be
accurate. However, it is particularly necessary to budget in any studies
of students for efforts to find fugitive students since otherwise the
extent of fluidity will be underreported.

IV.8: Questionnaire Studies of Employers

The principal value of collecting information from employers is to learn
more about their demand for PSOE (see II.1), particularly the conditions
under which they hire form community colleges, the value they place on
credentials including certificates and associate's degrees, licenses,
and industry-generated credentials compared to other dimensions of
qualifications including experience (see II.1 and III.1 on alternative
credentials). An additional purpose is to determine the ways in which
local institutions involve employers (see also II.1).

The issues of getting adequate response rates from employers are perhaps
as difficult as they are from students, though some researchers have
solved this problem (e.g., Osterman 1994). And all the problems with
force-choice questionnaires are applicable here too. One alternative
would be to include interviews with employers in the community case
studies (IV. 5), in order to get their perspectives on the variety of
programs within the community.

IV.9: Reanalyses of National Data

As I have stated in reviewing the existing literature (Grubb 1999b), any
subsequent analyses of national data ought to focus on several issues
that are currently unclear: the value of small amounts of course-taking,
where fields of study and student intentions need to be disentangled;
the role of local labor markets and employment conditions on the returns
to sub-baccalaureate education; and the timing of benefits, since it is
unclear whether the employment benefits of SOE materialize relatively
soon after completion or whether they take several years to develop.
NAVE should first determine the appropriateness of national data sets to
answering these questions before commission any further research.

IV.10: Reanalyses of State UI Data

NAVE should consider reanalyzing the UI wage record data already
collected by several states, in order to generate more sophisticated
multivariate analyses. As far as I have been able to ascertain, the
states with the most advanced data systems that are candidates for such
reanalysis include California, Texas, Florida, and Washington; a brief
investigation should be able to determine whether other states have
well-developed UI data. One purpose of reanalyses is to learn more about
the conditions under which programs are effective (see II.6), and
another is to provide some guidance to states in the analysis and
presentation of their data.

An outlier study would begin with an analysis of one state's data,
identify especially effective colleges, and then carry out local case
studies (parallel to those in IV.4) to identify possible reasons for
their success. A sample of perhaps three to five colleges in such a
state might generate enough information, as well as adding to case study
results.

IV.11: Reanalyses of Local Data

Some community colleges have collected relatively sophisticated data on
their own students and programs, and can use these data to examine the
effectiveness of local practices including the integration of academic
and occupational education, developmental education, tech-prep
activities, and the like. The local case studies should probe for such
local data, and NAVE could then either rely on local analyses or analyze
local data further. These data suffer the obvious problem that they are
inconsistent from college to college, and therefore cannot be used to
develop state or national patterns, but they do illustrate what has been
possible for local institutions.

An alternative is to conduct a more active search for such local data by
using networks of institutional researchers in community colleges. In
general, local data has been underutilized: many colleges collect more
data than they ever examine, and even when they are analyzed the results
are not widely known either inside or outside of individual colleges.
One possibility would be for NAVE to fund the national organization of
community college researchers, the National Council for Research and
Planning, to identify local results and data sets appropriate for
questions about PSOE and Perkins. This would have the added advantage of
engaging institutional researchers and their organization in both
federal policy and PSOE issues.

IV.12: Other Special Analyses

The analyses outlines above each involve several distinct question. In
addition, there are several single-purpose studies that need to be
carried out for specific issues raised in Sections I to III. These
include the following:

a\. The patterns of federal funding to PSOE through grants, loans, and
tax credits is one part of the larger question of how federal funds
support PSOE (see I.4). This requires data from different sources, some
of which describe student aid and other (like IRS data) could help
estimate the value of tax credits to PSOE.

b\. The results of alternative approaches to the sub-state allocation of
Perkins funds would require a simulation model (see I.6).

c\. Identifying different types of postsecondary institutions providing
PSOE would be helped by inquiring about such institutions from each
state, and finding state information about purposes, enrollments,
funding, and data on outcomes available.

d\. The analysis of industry-generated credentials and standards can be
accomplished in part through local case studies (IV.4), surveys of PSOE
faculty (IV.6), and surveys or interviews with employers (IV.8).
However, these are likely to miss many types of IGCs and IGSs. An
alternative is first to identify several sectors that have IGCs and
IOGSs, examine the structure of these standards, and then begin to
investigate their effects at the local level. For example, the
automotive program standards developed by NATEF could be selected; then
the relevant questions include which community colleges have adopted
NATEF standards and why, which employers require NATEF standards, and
whether instructors have modified their teaching in response to these
standards where they are widely used. Some of these questions can be
investigated through local case studies and questionnaires of faculty
and employers---though such responses require a focus on automotive
trades, and there are many other sectors and IGCs to investigate. In
this alternative, NAVE could identify a small number of
industry-generated standards---perhaps 5 to 10---and investigate their
effects through special studies, rather than relying on other forms of
data collection. It's also possible that this could be done in
conjunction with the National Skill Standards Board. This would help
understand the changing conditions of PSOE and help in defining who
"completers" are. In addition, a better understanding of the conditions
under which employer-driven skill standards emerge and thrive could
government groups like the NSSB understand how to create other kinds of
skill standards.

e\. Each of the support services identified in Section II.3 merits its
own analysis. While each of them can to some extent be addressed through
questionnaires (IV.3) and local case studies (see IV.4), and while
student surveys (IV.7) can generate other information on the use and
usefulness of existing services, these approaches won't generate much
information if, for example, the institutions selected for case studies
don't happen to have interesting approaches to remediation or
counseling. An alternative approach would be to supplement information
from case studies and questionnaires---which yield information on the
scope of conventional practices---with a more concerted effort to
identify conceptions and instances of good practice for each of these
support services.

f\. Finally, the ultimate question posed in Section III.4---the issue of
what the federal role in PSOE could be---would certainly benefit from
various studies including the analysis of what the effects of Perkins
III have been (the topics in Section I), but it also requires several
distinct activities. The development of an options paper based on
consideration of fiscal federalism and the history of other federal
programs is one, for example; a consensus-generating meeting would be
another; international comparisons might also be valuable and they are
quite different from any of the research activities presented so far.
Thus NAVE should consider supporting several activities specifically to
address this difficult normative question.

V. Some Priorities: A Personal View

In this review of PSOE research, I have presented far too many topics
for NAVE to support over the next three years. I presume that the
process of selecting topics will be one in which the Independent
Advisory Panel and others within government choose among these topics
based on their own conception of what NAVE should accomplish and which
issues are most important. And I hope that other topics can become part
of a broader agenda on PSOE that can be carried out by OERI and its
grants program, by the National Center on Postsecondary Improvement
sponsored by OERI, by the new National Center for Research in Vocational
Education, and by other federal and state research efforts.

However, I also offer my own priorities, with some brief justification.
In rough order they are as follows:

1\. NAVE needs to assess a few practices supported by Perkins to
determine how effective they have been. These include the current
development of accountability and performance measures, and the efforts
to broaden PSOE by requiring the incorporation and integration of
academic competencies.[^25] In contrast, I see no reason to study tech
prep yet one more time; the findings aren't going to change. In
examining these Perkins requirements, questionnaire surveys to local
officials are pointless without case studies to help interpret them, so
I would recommend more case studies and a shorter general questionnaire
rather than relying (as the previous NAVE did) wholly on long,
multi-part questionnaires.

2\. No doubt NAVE must consider the question of where the money goes
(see I.4). But this is a relatively uninformative counting exercise
without some kind of context about the alternative resources available.
Therefore, this exercise should be accompanied by (a) a fuller effort to
determine where other sources of federal funds go, and why (if I am
right) there are proportionately so few of them in community colleges
and other sub-baccalaureate institutions; and (b) determining how the
patterns of state resources affect the process of allocating federal
funds (from III.3).

3\. The study of who counts as a completer is among the most critical
issues facing PSOE, and involves both student issues and the apparent
growth of new kinds of credentials, including licenses and
industry-generated credentials. This ought to be a high priority for
NAVE; several of the specific research activities undertaken for this,
including surveys and interviews with students, will shed light on other
questions including elements of the "new fluidity."

4\. There are at least two powerful reasons to undertake a limited
number of community case studies: to understand the planning now taking
place with WIA, welfare, and other work force development programs (see
II.5 on linkages with other programs); and to identify the institutions
and the issues involved in the "new fluidity" (III.2). Thus a nested
design of questionnaires to a large number of PSOE providers, a smaller
number of local case studies, and a still smaller number of community
case studies would address a large number of issues.

5\. Of the multiple dimensions of effective practice I presented in
Section II, several will be covered in research I have already outlined:
the nature of licenses and industry-generated credentials in examining
the issue of completers, the use of data in analyzing accountability,
and linkages to other programs in community case studies. For me, the
are two other major candidates for NAVE research in this section: the
pedagogy of PSOE (including the pedagogy of work-based learning), since
this topic has been so rarely addressed; and further analysis of state
UI data systems, including the possibility of an outlier study.

6\. The final question---what the federal role in PSOE should be---might
be undertaken by NAVE staff, but certain supporting
activities---particularly an issues and options paper, and a
consensus-generating activity with various constituencies
present---would be helpful in formulating the alternatives.

The issues surrounding PSOE---including the need for greater visibility
and stature, the requirements for additional research, and the
development of a compelling vision for federal policy---are much larger
than a single research exercise like NAVE can address. But NAVE can at
least help define what these issues are, and therefore help create an
agenda for the twenty-first century.

References

Achtenhagen, F., and Grubb, W.N. Forthcoming. "Vocational and
Occupational Education: Pedagogical Complexity, Institutional
Indifference." In *Handbook of Research on Teaching.* 4th ed. Ed. V.
Richardson. Washington, DC: American Educational Research Association.

Badway, N. 1998. *The Politics of Compliance: Federal Policy for
Curriculum Reform in Community Colleges.* Ph.D. dissertation, University
of California, Berkeley.

Beatty-Guenther, P. 1994. *Retention Strategies at Community Colleges:
Diffusion, Legitimacy, and Effectiveness.* Ph.D. dissertation, Berkeley:
School of Education, University of California, Berkeley.

Bragg, D. 1996. *Perspectives of Urban Educators, Students, and
Employers Toward Tech Prep Student Outcomes.* Berkeley: National Center
for Research in Vocational Education.

Bragg, D., Layton, J., and Hammons, F. September 1994. *Tech Prep
Implementation in the United States: Promising Trends and Lingering
Challenges.* Berkeley: National Center for Research in Vocational
Education, University of California.

Bragg, D., Hamm, R., and Tinkle, K. 1995. *Work-Based Learning in
Two-Year Colleges in the United States.* Berkeley: National Center for
Research in Vocational Education, University of California.

Brewer, D., and Gray, M. October 1997. *Connecting College and Community
in the New Economy? An Analysis of Community College Faculty-Labor
Market Linkages.* MDS-1084. Berkeley: National Center for Research in
Vocational Education, University of California.

Cohen, A.M., and Brawer, F.B. 1989. *The American Community College.*
2nd ed. San Francisco: Jossey-Bass.

Committee for Economic Development. 1985. *Investing in Our Children:
Business and the Public Schools.* New York: Author.

Dougherty, K. 1994. *The Contradictory College: The Conflicting Origins,
Impacts, and Futures of the Community College.* Albany: State University
of New York Press.

Grubb, W.N. 1996a. *Learning to Work: The Case for Re-Integrating Job
Training and Education*. New York: Russell Sage Foundation.

Grubb, W.N. 1996b. *Working in the Middle: Strengthening Education and
Training for the Mid-Skilled Labor Force*. San Francisco: Jossey-Bass.

Grubb, W.N. 1997. "The Returns to Education in the Sub-Baccalaureate
Labor Market, 1984--1990." *Economics of Education Review* 16 (3),
231--246.

Grubb, W.N. February 1999. *Lessons from Education and Training for
Youth: Five Precepts.* Paper presented at the OECD Conference on
"Preparing Youth for the 21st Century: Policy Lessons from the Past Two
Decades," Washington, DC.

Grubb, W.N. Forthcoming. *Learning and Earning in the Middle: The
Economic Benefits of Sub-Baccalaureate Education.* New York: Community
College Research Center, Teachers College, Columbia University.

Grubb, W.N., and Associates. 1999. *Honored But Invisible: An Inside
Look at Teaching in Community Colleges.* New York and London: Routledge.

Grubb, W.N., Badway, N., Bell, D., and Kraskouskas, E. 1996. *Community
College Innovation in Workforce Preparation: Curriculum Integration and
Tech Prep.* Mission Viejo, CA and Berkeley: League for Innovation in the
Community College, NCRVE, and National Council for Occupational
Education.

Grubb, W.N., Badway, N., Bell, D., King, C., Herr, J., Prince, H.,
Kazis, R., Hicks, L., and Taylor, J. 1999. *Toward Order From Chaos:
State Efforts to Reform Workforce Development Systems.* Berkeley, CA:
National Center for Research in Vocational Education.

Grubb, W.N., and Kalman, J. November 1994. "Relearning to Earn: The Role
of Remediation in Vocational Education and Job Training." *American
Journal of Education* 103 (1): 54--93.

Grubb, W.N., and Kraskouskas, E. 1992. *A Time to Every Purpose:
Integrating Academic and Occupational Education in Community Colleges
and Technical Institutes*. Berkeley: National Center for Research in
Vocational Education, University of California, Berkeley.

Grubb, W.N., and McDonnell, L.M. 1991. *Local Systems of Vocational
Education and Job Training: Diversity, Interdependence, and
Effectiveness*. Santa Monica and Berkeley: RAND and National Center for
Research in Vocational Education, University of California, Berkeley.

Grubb, W.N., and McDonnell, L. 1995. "Combating Program Fragmentation:
Local Systems of Vocational Education and Job Training." *Journal of
Policy Analysis and Management* 15 (2): 252--270.

Grubb, W.N., and Ryan, P. Forthcoming. *The Roles of Evaluation for
Vocational Education and Training: Plain Talk on the Field of Dreams*.
London and Geneva: Kogan Paul and International Labour Office.

Grubb, W.N., and Stasz, C. September 1998; original 1993. "Integrating
Academic and Vocational Education: Progress under the Carl Perkins
Amendments of 1990." In *Educational Reform and Vocational Education.*
Ed. A Milne. Washington DC.: Office of Educational Research and
Improvement, U.S. Department of Education.

Grubb, W.N., and Stern, D. January 1989. "Long Time A Comin': Options
for Federal Financing of Postsecondary Vocational Education."
Unpublished manuscript, National Assessment of Vocational Education.
Berkeley: School of Education, University of California, Berkeley.

Grubb, W.N., and Tuma, J. Spring 1991. "Who Gets Student Aid? Variations
in Access to Aid." *Review of Higher Education* 14 (3): 359--381.

Harris, N.C., and Grede, J.F. 1977. *Career Education and Colleges: A
Guide for Planning Two- and Four-Year Occupational Programs for
Successful Employment.* San Francisco: Jossey-Bass.

Hershey, A., Silverberg, M, Haimson, J., Hudis, P, and Jackson, R.
September 1998. *Expanding Options for Students: Report to Congress on
the National Evaluation of the School-to-Work Implementation.*
Princeton, NJ: Mathematica Policy Research.

Hershey, A., Silverberg, M., Owens, T., and Hulsey, L. 1998. *Focus for
the Future: The Final Report of the National Tech-Prep Evaluation.*
Princeton, NJ: Mathematica Policy Research.

Hollinger, D., and Harvey, J. July 1994. "Coordinating Vocational
Education and Federal Job-training Programs." In *Final Report to
Congress: Vol. 1. Funding and Administration Issues*. Eds. L. Muraskin,
D. Hollinger, and J. Harvey. Washington, DC: U.S. Department of
Education.

Huber, M. 1998. *Community College Faculty Attitudes and Trends, 1997.*
Menlo Park: Carnegie Foundation for the Advancement of Teaching.

King, C., Lawson, L., Olson, J., Trott, C., and Baj, J. 1998. "Training
Success Stories for Adults and Out-of-School Youth: Past Effects and
Lessons for the Future." In *Improving the Odds: Increasing the
Effectiveness of Publicly-Funded Training*. Eds. Burt Barnow and
Christopher King. Washington D.C.: Urban Institute Press, forthcoming.

Mayer, D.P. Spring 1999. "Measuring Instructional Practices: Can
Policymakers Trust Survey Data?" *Educational Evaluation and Policy
Analysis* 21 (1): 29--46.

McDonnell, L., and Grubb, W.N. April 1991. *Education and Training for
Work: The Policy Instruments and the Institutions.* Berkeley and Santa
Monica, CA: National Center for Research in Vocational Education and the
RAND Corporation.

Osterman, P. 1994. "How Common is Workplace Transformation and Who
Adopts It?" *Industrial and Labor Relations Review* 47 (2): 173--188.

Perrin, D. September 1998. *Curriculum and Pedagogy to Integrate
Occupational and Academic Instruction in the Community College:
Implications for Faculty Development.* Occasional Paper. New York:
Community College Research Center, Teachers College, Columbia
University.

Pinsky, R. 1994. *The Inferno of Dante.* New York: Farrar, Straus, and
Giroux.

Richardson, R.C., Jr., Fisk, E.C., and Okun, M.A. 1983. *Literacy in the
Open-Access College*. San Francisco: Jossey-Bass.

Roe, E. 1994. *Narrative Policy Analysis: Theory and Practice.* Durham,
NC, and London: Duke University Press.

Schneider, B., and Stevenson, D. 1999. *The Ambitious Generation:
America's Teenagers, Motivated But Directionless.* New Haven: Yale
University Press.

Smith, V. June 1993. "Phantom Students: Student Mobility and General
Education." *AAHE Bulletin* 45 (10): 10--13, 7.

Stasz, C. May 1999. *Assessing the Quality of Vocational Education in
High Schools.* Paper presented at the meeting of the Independent
Advisory Panel, National Assessment of Vocational Education, Washington,
DC.

Tuma, J. April 1993. *Patterns of Enrollment in Postsecondary Vocational
and Academic Education*. Prepared for the National Assessment of
Vocational Education. Berkeley, CA: MPR Associates, Inc.

Walker G. June 1997. "Out of School and Unemployed: Principles for More
Effective Policies and Programs." In *A Generation of Challenge:
Pathways to Success for Urban Youth*. A. Sum et al. (Policy Issues
Monograph 97-03), pp. 73--86. Baltimore: Sar Levitan Center for Social
Policy Studies.

Wills, J.L., and Lurie, I. 1993. *Industry-Driven Skill Standards
Systems in the United States Vol. III* . Washington, DC: Institute for
Educational Leadership, Center for Workforce Development.

[^1]: I have been immeasurably helped in preparing this paper by a
    meeting of the NAVE Independent Advisory Panel on May 6--7, 1999; by
    a meeting with individuals at the American Association of Community
    Colleges, especially Jim McKenney, Kent Phillippe, and Margaret
    Rivera; by the participants' comments in a meeting held at the April
    1999 AACC meetings by Rick Hernandez and Jon Weintraub of the U.S.
    Department of Education; by meetings with the board of the
    California Community College Association of Occupational Educators
    and subsequent discussions with occupational deans in both the San
    Francisco area and the Los Angeles area; and by discussions with
    Norena Badway of the Community College Cooperative, NCRVE; Jim
    Jacobs of Macomb Community College; Richard Fonte of Austin
    Community College; and David Goodwin and Elizabeth Warner of OUS.

[^2]: In this paper I won't get into the federal limitation of
    vocational education to education below the baccalaureate. This
    constraint prevents seeing vocational and professional education at
    the baccalaureate and graduate level as continuous with one
    another---even though the pedagogical and policy issues are quite
    similar, and the analysis of vocational (or professional) education
    in four-year college is as neglected as PSOE.

[^3]: This is based on 1992--93 National Postsecondary Student Aid
    Survey data; see Tuma (1993), Table 2.1.

[^4]: See especially Section III.2, on the role of PSOE in the "new
    fluidity" that defines current postsecondary education and training.

[^5]: I have summarized much of my own work, carried out largely with
    the support of the National Center for Research in Vocational
    Education, in Grubb (1996b). One has to go back to Harris and Grede
    (1977) to find a comprehensive volume on PSOE; most reviews of
    community colleges, like Cohen and Brawer (1989) or Dougherty
    (1994), devote relatively little attention to PSOE. See also the
    *Journal of Studies in Technical Careers* published by the National
    Commission on Occupational Education.

[^6]: Readers will doubtless remember that the eighth circle of hell is
    reserved for those committing sins of fraud, including false
    counselors. See Cantos 26 to 28 of Dante's *Inferno,* especially the
    wonderful translation by Pinsky (1994) illustrated by Michael Mazur.

[^7]: I've been impressed with the extent to which policy in many
    countries is driven by narratives, or widely accepted and simple
    "stories" about why certain programs are worthwhile. The creation of
    such narratives typically takes a considerable period of time and
    many different participants; once widely accepted, policy narratives
    are resistant to change, and empirical evidence---the kinds of
    results that academics can generate---is not usually enough to
    modify or complicate a policy narrative. See, for example, Roe
    (1994).

[^8]: See Grubb and Stasz (1998), Table 4.31, and the discussion of
    postsecondary integration on pp. 181--186. These figures might be
    accurate if each institution reporting integration practices has
    only one or two instances of it, but this is still misleading about
    the prevalence of integration practices. The postsecondary
    questionnaire was devised by me, using a typology of approaches to
    postsecondary integration first presented in Grubb and Kraskouskas
    (1992). My recommendation to complement questionnaire results with
    local case studies was not implemented, perhaps because of lack of
    funding.

[^9]: As if! Most observers feel that these tax credits were merely
    vehicles for middle-class tax relief, in which case the rational
    issues I pose about the distribution of these funds are simply
    irrelevant.

[^10]: This is not uniformly true; several of the technical institutes
    in Wisconsin have started transfer programs, and some faculty there
    predicted that the higher-status academic role would begin to
    diminish the occupational programs.

[^11]: In Minnesota technical institutes and community colleges were
    combined; in South Carolina technical institutes were converted into
    comprehensive community colleges; there are apparently some
    initiatives in Louisiana to convert a welter of technical institutes
    into a more coherent system of comprehensive community colleges.

[^12]: On the ineffectiveness of short-term job training programs, see
    Grubb (1996a).

[^13]: For fuller development of these five criteria, see Grubb (1999a),
    Grubb and Ryan (1999), or Grubb et al. (1999). It's surprising to
    see how different such lists of principles can be; for a very
    different set of principles, focusing on how students feel (e.g.,
    "each young person needs to feel that at least one adult has a
    strong stake and interest in his or her labor market success"), see
    Walker (1997).

[^14]: The source of my nastiest statements include Grubb and Associates
    (1999), especially Ch. 5 on teaching in remedial/developmental
    classes; and Grubb and Kalman (1994).

[^15]: WIA requires states to complete state plans between April 1,
    1999, and April 1, 2000, with all states implementing WIA by July 1,
    2000. Quite apart from the fact that many states may be late with
    their plans, this means that the year 2000--2001 will be the first
    year of full implementation of WIA, and we can anticipate that this
    will be a year of transition and confusion. But this is the second
    and final year of NAVE's data collection, so that NAVE at best will
    be able to learn about plans in progress.

[^16]: The technique of community case studies underlies the NCRVE
    research reported in Grubb and McDonnell (1991 or 1995), for
    example.

[^17]: Even the receipt of an associate's degree or certificate may be
    ambiguous. In California, some colleges require students to apply
    for these credentials, while others grant them automatically when a
    student has accumulated sufficient credits. Rates of completion are
    obviously higher in the latter institutions.

[^18]: In my own interviews with students, reported briefly in Grubb
    (1996b, Ch. 2), it became clear that many students who reported
    wanting to transfer had no idea about what that might mean. These
    were effectively "experimenters" who reported transfer as their
    intention because of the limitations of forced-choice
    questionnaires.

[^19]: For example, Richard Fonte, the president of Austin Community
    College, has suggested that Austin, Texas, and Portland, Oregon,
    might be appropriate test cases because they have been so
    substantially affected by IGCs in the information technology and
    computing areas, and because they are in relatively bounded labor
    markets.

[^20]: The only effort I know to examine IGCs and IGSs is the listing of
    such efforts in Wills and Lurie (1993), but this does not go beyond
    the attempt to describe the various industry groups that have taken
    this approach.

[^21]: Note that the baccalaureate degree may also be subject to the
    "new fluidity"; see Smith (1993) on the damage the practice of
    taking courses in multiple institutions causes to general education.
    However, the baccalaureate degree remains a sturdy image of how to
    provide students with extensive choice---of colleges, majors, and
    specific general education requirements---within an institutional
    structure of requirements, electives, and progression through four
    years or so.

[^22]: A now-dated effort to do this was undertaken for the NAVE in the
    late eighties, together with a section on the options for federal
    legislation; see Grubb and Stern (1999).

[^23]: To put the issue in different terms, the process of creating
    broad policy narratives is complex and not necessarily rational. See
    the comments in footnote 7.

[^24]: In the interviews with students reported in Grubb (1996b, Ch. 2)
    I tried to collect such school and work histories, with mixed
    success; students don't remember well those activities that have had
    little importance for them. I also tried to put together such a
    history of activities with the NLS72 data, which has a great deal of
    information both about employment and education over time. However,
    missing data made this virtually impossible, and I despair of using
    any national data set for this purpose. An alternative would be to
    replicate for postsecondary students the Sloan Study of Youth and
    Social Development, first reported in Schneider and Stevenson
    (1999), which followed students from sixth, eighth, tenth, and
    twelfth grades over a five-year period.

[^25]: The allowance of "all aspects of the industry" is part of
    developing a broader conception of occupational education and should
    be included here.