Archived Information

Improving State Accountability Systems

for Postsecondary Vocational Education

Robert G. Sheets

Business and Industry Services

### Northern Illinois University

April 2002

### 

### 

This paper was prepared for the Office of Vocational and Adult
Education, U.S. Department of Education pursuant to contract no.
ED-99-CO-0160. The findings and opinions expressed in this paper do not
necessarily reflect the position or policies of the U.S. Department of
Education.

Improving State Accountability Systems for

Postsecondary Vocational Education

Robert G. Sheets

### 

### *Introduction*

Since the 1980s, state community and technical college boards and
agencies and institutions have experienced a major increase in federal
and state accountability requirements. This increase has been
accompanied by a shift in emphasis from access to performance. This new
environment is prompting state boards and agencies and institutions to
develop new performance indicator systems for reporting and managing
continuous improvement for all major functions and programs, including
vocational education.

The Carl D. Perkins Vocational and Applied Technology Education
Amendments of 1998 (Perkins III) have been a major driving force in
creating this new accountability environment and in shaping state and
institutional responses to it. The development of the Perkins III
accountability system for postsecondary vocational education marked the
first major federal-state effort to develop and fully implement a
national framework for state accountability systems for vocational
education in the United States. This effort broke new ground and
improved the measurement and reporting of secondary and postsecondary
student outcomes in vocational education.

However, this effort also highlighted some major issues that should
frame future discussions and efforts to improve postsecondary
accountability systems. One major issue is whether the Perkins III core
indicators are comprehensive and valid indicators for measuring the
performance of postsecondary vocational education and whether states
have the capacity to collect and report high quality data for these
measures. A second issue is whether the Perkins III accountability
requirements are creating unnecessary burden on states and institutions
and how to improve their value and lower their costs. A third major
issue is whether there should be more consistency between these measures
and other federal, state, and institutional performance measures (OVAE,
2000b). Should the Perkins III core indicators be designed to fit within
a larger postsecondary education framework that integrates vocational
education measures with other postsecondary education and workforce
development measures?

The purpose of this paper is to make recommendations for developing the
next generation of state accountability systems for postsecondary
vocational education. This paper focuses on the need to: (1) make
improvements in the core indicators for postsecondary vocational
education, (2) reduce the burden and improve the value of Perkins
accountability requirements, and (3) integrate future federal vocational

education measures into a comprehensive institution-centered framework
that integrates postsecondary education and workforce development
performance indicators. This paper also makes recommendations for the
federal role in accountability and how the Office of Vocational and
Adult Education (OVAE) can play a catalyst role in improving state and
institutional accountability systems for postsecondary education,
including vocational education.

This paper first summarizes the major trends in the postsecondary
accountability environment at the national and state levels and how
community and technical colleges are responding by creating performance
indicator frameworks. The paper then reviews the major accomplishments
and remaining problems and issues in the Perkins III effort and makes
recommendations on how to build from these efforts. Finally, the paper
makes recommendations on developing and supporting core indicators for
postsecondary vocational education and how they can be incorporated into
broader institution-centered accountability systems for managing
performance improvement.

## 

## The Changing Postsecondary Accountability Environment for  {#the-changing-postsecondary-accountability-environment-for .OVAE}

## Community and Technical Colleges {#community-and-technical-colleges .OVAE}

What are the major changes in the postsecondary accountability
environment that are shaping the development and implementation of new
accountability systems at the national, state, and institutional levels?
How are state community and technical college systems and institutions
responding? What are the implications for future efforts in developing
federal accountability systems for postsecondary vocational education?

## Quality Assurance: From Self-Regulation to External Accountability Systems {#quality-assurance-from-self-regulation-to-external-accountability-systems .OVAE}

Throughout most of the 20^th^ century, federal and state governments
looked to higher educational institutions to define their own mission
and goals and to define the criteria for quality in higher education
services (Bogue and Saunders, 1992). Higher educational institutions
were also given the authority to regulate themselves by developing and
managing their own quality assurance systems through internal management
systems and participating in national and regional accreditation
systems. Leading higher educational institutions and federal and state
governments promoted and supported the establishment of national and
regional accreditation organizations to assure the quality of
postsecondary education.

These self-regulation systems resulted in the development of internally
focused quality assurance systems that were based on missions and goals
defined by administrators and faculty members with only limited input
from external stakeholders and customers. They also lead to the
development of internally focused "quality indicators" that specified
the organizational capabilities, conditions, practices, and resources
necessary for providing high quality postsecondary educational services
to students and fulfilling an institution's mission and goals. These
quality indicators provided the basis for government-endorsed,
peer-controlled accreditation systems that used institutional self-study
and analysis followed by peer review and validation coordinated through
national and regional accreditation organizations.

Starting in the 1970s and 1980s, federal and state governments and other
external stakeholders took a more active role in ensuring the quality of
postsecondary education in the United States. The federal government
added new reporting requirements and set performance requirements for
student grant and loan programs. States expanded the roles of higher
education boards and commissions in ensuring greater accountability.
Both federal and state governments established new external
accountability systems to ensure that higher education was addressing a
broader set of public and consumer expectations for postsecondary
education and were measuring and monitoring the fulfillment of these
expectations. These new systems did not replace accreditation. Rather,
they represented an additional layer of accountability.

As noted by Levine (1992), this shift was prompted not by a
well-documented "failure of performance" which was the driver in the
K-12 accountability movement. Rather, it was a more generalized "crisis
of confidence" and a growing recognition that federal and state
governments were not taking an active role in defining the public
interest in higher education and managing their sizeable investments.
This shift was prompted by growing distrust of higher education and the
perceived need to get a handle on the costs, productivity and
effectiveness of higher education during a period of economic downturn
and shrinking federal and state resources. There was a widespread and
growing perception that traditional mechanisms for self-regulation and
quality assurance were in disarray and could not be trusted.

These federal and state initiatives to establish external accountability
systems prompted a major shift in national and regional accreditation
organizations to focus more attention on external stakeholder and
customer requirements. They also prompted a shift from internal "closed
system" to external "open-system" models for quality assurance within
higher educational institutions, including community and technical
colleges (Alfred, 1997).

## Performance Indicators: Expanding the Focus to Student Outcomes {#performance-indicators-expanding-the-focus-to-student-outcomes .OVAE}

During the 1970s and 1980s, national and state accountability
initiatives focused on developing comprehensive performance indicator
systems to monitor the effectiveness of higher education (Borden and
Bottrell, 1994). The early performance indicator systems focused on
access, productivity, and quality with only limited attention to student
outcomes. These initiatives focused on increasing access to higher
education, especially minorities and underserved populations and in
controlling costs and increasing productivity. Service output and cost
measurement focused attention on how dollars were spent and justified
the need for additional spending based on the quantity of services
(e.g., units of instruction) provided and the additional costs of these
services. During this same period, many states established institutional
and program effectiveness reviews that used "program quality" indicators
along with market demand, student enrollment, and cost information to
improve the quality and mix of postsecondary programs. These program
quality indicators were similar to the quality criteria established by
national and regional accreditation organizations.

During the 1990s, national and state initiatives expanded the focus to
measuring the performance of postsecondary educational institutions
based in part on student outcome indicators. Higher educational
institutions were increasingly held accountable not only for providing
access to high quality programs, but also for achieving results
important to students and external stakeholders. Two major types of
student outcomes were learning outcomes and economic outcomes.

**Learning Outcomes: Skill and Credential Attainment**. Most early
federal and state initiatives focused major attention on the success of
students in attaining postsecondary degrees. Some state performance
indicator systems also reported course completion rates with the
assumption that degree or course completion was a valid and reliable
indicator of success in achieving learning outcomes. Starting in the
1980s, this assumption was called into question. The growing distrust of
higher education was heightened by national studies and reports on
lowered standards and grade inflation and the skill deficiencies of
college graduates. Public and private leaders called for major reforms
in postsecondary assessment and credentialing (National Governor's
Association, 2000). This resulted in major national and state
initiatives to improve postsecondary assessment and credentialing
systems so that they can provide valid and reliable documentation of
student learning outcomes. This also has prompted national and regional
accreditation organizations to focus on improving student assessment
systems and promoting new "outcome-based accreditation" models for
four-year and two-year higher educational institutions. This remains a
major challenge for postsecondary education as exemplified by the recent
National Report Card on Higher Education that gave all states a grade of
"incomplete" on their performance in student learning (National Center
for Public Policy and Higher Education, 2000). This is also exemplified
by national initiatives by higher education and business organizations
to improve the quality of credentials including postsecondary education
degrees and certificates (National Alliance of Business, 2001).

**Economic Outcomes: Employment and Earnings**. During the 1980s and
1990s, the focus of higher education accountability efforts expanded
beyond learning outcomes to include economic returns to students
attaining postsecondary degrees and credentials. The beginnings of this
shift can be traced to the early 1980s when postsecondary educational
institutions justified the need for higher levels of federal and state
investment because of their contributions to state and local economic
development and the increased earnings of American workers (Ewell,
1994). This also can be traced to efforts by postsecondary education
representatives to justify increased tuition costs by arguing that
postsecondary education remained a "high-value" investment for students
because of the increasing economic returns to postsecondary education in
the form of higher wages and earnings. With increased federal and state
funding and greater flexibility in increasing tuition costs, higher
education promised to promote state and local economic development and
raise the earnings of college graduates. In the 1990s, public and
private leaders, including governors and state legislators, began asking
for evidence that postsecondary education was delivering on these
promises. In response, state higher educational agencies and boards
launched initiatives to address the economic returns to postsecondary
education. These initiatives included the development of new performance
indicators and systems to collect information on the employment and
earnings of students receiving postsecondary degrees and credentials
(Mundhenk, 2000; Seppanen, 1995). These efforts were focused on the
professional degree programs of four-year colleges and the career and
technical programs at two-year community and technical colleges.

## Increased Requirements for Consumer Information and Feedback  {#increased-requirements-for-consumer-information-and-feedback .OVAE}

During the 1980s and 1990s, most industries faced a consumer revolution.
Consumers wanted more power in their purchasing decisions by having more
information about products and services and the choices they were given
in the marketplace. Consumer organizations and businesses responded by
gathering and reporting comparative information on products and services
and developed new materials and services to advise consumers.

As the costs of higher education continued to rise throughout the 1980s
and 1990s and the value of higher education degrees increased in the
labor market, this consumer revolution spilled over into the higher
education marketplace. Parents, students, and public officials began
demanding to know more about what they were getting for their money and
they wanted the information to make comparisons between institutions and
programs to improve their decisions. In response, federal and state
governments focused on improving consumer information to help students
chose colleges and universities with proven track records.

**Consumer Information Systems**. Federal and state governments launched
major consumer information initiatives to respond to this growing demand
for consumer information in higher education. Federal higher education
legislation established "student-right-to-know" requirements for higher
educational institutions. Federal agencies and departments converted
national reporting systems, such as IPEDS, into consumer information
systems that provided comparative information on colleges and
universities. In addition, the Workforce Investment Act (WIA)
established strong consumer information requirements for training
providers, which included many community and technical colleges. These
government initiatives developed reporting and consumer information
systems to provide accurate and reliable data in a wide variety of areas
including program completion and employment and earnings.

In addition, private organizations including leading magazines, such as
U.S. News and World Report, began compiling and distributing comparative
information on the "quality" and "value" of higher educational
institutions and specific degree programs. These private efforts have
lead to the development of new measures for the quality and performance
of higher education institutions.

**Customer Feedback: Measuring and Reporting Customer Satisfaction**.
This consumer revolution in conjunction with a related quality movement
in higher education has lead to more emphasis on measuring and reporting
customer satisfaction. The higher education community is debating how to
measure and use student feedback (Kulik, 2001) and overall customer
satisfaction in performance reporting and internal improvement efforts
similar to leading public and private sector models such as Kaplan and
Norton's Balanced Scorecard and the Baldrige framework. This is also
being debated in the implementation of WIA requirements for measuring
customer satisfaction.

## From Performance Reporting to Performance Management: Financial Incentives and Continuous Improvement  {#from-performance-reporting-to-performance-management-financial-incentives-and-continuous-improvement .OVAE}

During the 1990s, federal and state agencies used their performance
indicators systems to not only report performance data but to manage
performance. This was done by providing financial incentives to promote
improvements and developing performance management requirements that
asked institutions to develop and manage improvement strategies and
report progress on these efforts.

Performance-Based Planning and Budgeting. Federal and state governments
are undertaking major initiatives to use financial incentives and the
budgeting process to drive performance improvement on a core set of
performance measures. The performance incentive requirements in the WIA
and Perkins III are only two examples of the use of performance
incentives in federal legislation and federal agencies. In addition,
states have been major players in using performance-based budgeting and
financial incentives (Liner et al., 2001). In particular, states have
taken major steps to use financial incentives and new approaches to
planning and budgeting to promote performance improvement at the system
and institutional levels in higher education including community and
technical colleges (Ashworth, 1994).

Performance Planning and Management. The Government Performance and
Results Act (GPRA) required federal agencies to develop strategic plans
that established performance goals and strategies and "results
frameworks" to manage improvement of performance over multiple years.
States also established similar strategic planning and performance
management requirements for agencies. These new federal and state
requirements have expanded the purpose of performance indicators systems
from reporting to overall performance management.

## The Response of Community and Technical College Systems: The Development of Performance Indicator Systems {#the-response-of-community-and-technical-college-systems-the-development-of-performance-indicator-systems .OVAE}

##  {#section-4 .OVAE}

State higher education boards, including community and technical college
boards, and colleges have responded to the new accountability
environment by developing performance indicators systems. These
indicator systems are designed to report performance to federal agencies
and state legislatures and manage improvement efforts at the state and
institutional levels.

State higher education boards have developed comprehensive performance
indicator systems for annual performance reports for state higher
education systems. Some indicators apply to both four-year and two-year
institutions (e.g., degree completion rates). Others are specifically
designed to address the unique missions and roles of community and
technical colleges. Community and technical college systems also have
responded to the new accountability environment by developing
comprehensive core indicator frameworks to measure and report
"organizational effectiveness." In 1994, the American Association of
Community Colleges held a national roundtable to share information on
leading practices in developing and using "core indicators of
effectiveness." This effort led to the development of national
benchmarks for core indicators (Alred et al., 1999).

The major performance areas and selected example measures from state and
institutional core indicator frameworks are summarized in Figure 1.
These performance indicator systems reflect the multiple and distinct
missions of community and technical colleges including their roles in:
(1) academic preparation and transfer to four-year colleges; (2) the
preparation for careers and employment; (3) adult, remedial and
developmental education; (4) economic development and business and
technology services; and (5) community development/service (Bailey and
Averianova, 1998).

These core indicator frameworks also reflect the highly diverse student
populations served by community and technical colleges and the wide
range of goals these students have in attending two-year colleges.
Community and technical colleges have a unique role in higher education
not only because of their different roles and functions but the students
they serve. They are widely recognized as providing open access to a
wide variety of non-traditional students who historically have not
participated in postsecondary education. These students have a wide
variety of goals that are different than traditional students. Many of
these non-traditional students only want to gain a specific skill, not
complete a program. Many of these students are already employed and are
interested in changing jobs or career advancement.

+-----------------------------------------------------------------------+
| ## Figure 1: Selected Indicators from State/Institutional Systems     |
|                                                                       |
| ## Student Access and Persistence                                     |
|                                                                       |
| -   Number and percent of minorities and other targeted populations   |
|     enrolling in postsecondary education                              |
|                                                                       |
| -   Student persistence (fall to fall enrollment)                     |
|                                                                       |
| ## Skill and Credential Attainment                                    |
|                                                                       |
| -   Skill attainment                                                  |
|                                                                       |
| -   Course completion                                                 |
|                                                                       |
| -   Degree/certificate attainment rate                                |
|                                                                       |
| -   Licensure/certification pass rates                                |
|                                                                       |
| ## Student Transition                                                 |
|                                                                       |
| -   Placement rate in employment                                      |
|                                                                       |
| -   Earnings at placement                                             |
|                                                                       |
| -   Transition to four-year colleges                                  |
|                                                                       |
| -   Retention at four-year colleges                                   |
|                                                                       |
| -   Grade point averages/course grades of transitioned students       |
|                                                                       |
| ## Customer Satisfaction                                              |
|                                                                       |
| -   Student goal attainment (self-reported)                           |
|                                                                       |
| -   Student satisfaction with services                                |
|                                                                       |
| -   Employer satisfaction with services                               |
+-----------------------------------------------------------------------+

Finally, these core indicator frameworks reflect the expanded definition
of the customers and the growing recognition of employers and industry
organizations as customers for economic and workforce development
services. As community and technical colleges are expanding their role
in economic and workforce development, they are increasingly addressing
the needs of employers and major industries in their service regions.

## Implications for Developing New Systems {#implications-for-developing-new-systems .OVAE}

##### 

##### What are the implications of this changing postsecondary accountability environment and the responses of community and technical colleges for future efforts to develop federal performance accountability systems for postsecondary vocational education? The first implication is that future federal efforts should attempt to integrate multiple federal performance requirements and should be coordinated with state and local efforts to develop comprehensive frameworks for postsecondary institutions that balance the multiple missions and customer requirements of community and technical colleges. The second implication is that future efforts should attempt to improve the integration of federal and state performance measures with consumer information requirements. The third implication is that the focus should shift from federal reporting to performance management with an emphasis on using performance information and financial information to manage improvement and improve the return on public investment at the federal, state, and local levels. 

## Building from the Perkins III Core Indicator Framework:  {#building-from-the-perkins-iii-core-indicator-framework .OVAE}

## Problems and Issues {#problems-and-issues .OVAE}

The development of the Perkins III accountability system for
postsecondary vocational education marked the first major federal-state
effort to develop and fully implement a national framework for state
accountability systems for vocational education in the United States.
Perkins III made a major shift from Perkins II in the definition of the
required indicators of performance moving from a gains to an attainment
focus. OVAE worked with states for over a year to develop a core
indicator framework that would provide the basis for addressing each of
the required core indicators of performance while at the same time
providing states with the flexibility to define their own measurement
approaches and build from current state data collection and measurement
systems.

This section first describes the Perkins III core indicator framework
and then addresses three major issues:

-   What should we be measuring and how do we improve state and
    institutional capacity for collecting high quality data?

-   How do we reduce the burden and improve the value of the Perkins III
    accountability system for states and institutions?

-   How do we improve the consistency and integration of federal and
    state core performance indicators for postsecondary education and
    workforce development?

## Perkins III Core Indicator Framework {#perkins-iii-core-indicator-framework .OVAE}

Perkins III defines four major core indicators of performance, which
provided the basis for developing the core indicator framework.

(1) Student attainment of challenging State established academic and
    vocational and technical skill proficiencies.

(2) Student attainment of a secondary school diploma or its recognized
    equivalent, a proficiency credential in conjunction with a secondary
    school diploma, or a postsecondary degree or credential.

(3) Placement in, retention in, and completion of postsecondary
    education or advanced training, placement in military service, or
    placement or retention in employment.

(4) Student participation in and completion of vocational and technical
    education programs that lead to non-traditional training and
    employment.

The Perkins III core indicator framework contains these core indicators
for both secondary and postsecondary education. As shown in Figure 2,
this framework established four core indicators and seven core
subindicators for postsecondary vocational education. These
subindicators are identified by a "P" next to each core subindicator.

The core indicator framework defines the performance measures and
measurement approaches for the fourteen subindicators. It also defines
the quality criteria for these measurement approaches. As defined in the
core indicator framework document (OVAE, 2000a), these are:

-   [Performance Measures]{.underline}. The definition of the
    performance measures for each subindicator including the definition
    of the numerator and denominator of each performance measure.

-   [Performance Measurement Approaches]{.underline}. The major state
    approaches for performance measurement for each performance measure.
    These approaches include assessment and data collection strategies.

-   [Quality Criteria for Performance Measurement]{.underline}. The
    quality criteria for performance measurement to ensure sufficient
    rigor and comparability of state performance measurement and
    reporting.

**Figure 2: Perkins III Core Indicators and Subindicators**

###### Core Indicator 1. Student Attainment

1S1: Secondary Academic Attainment

1S2: Secondary Vocational and Technical Skill Attainment

1P1: Postsecondary Academic Attainment

1P2: Postsecondary Vocational and Technical Skill Attainment

###### Core Indicator 2. Credential Attainment

2S1: Secondary Completion

2S2: Proficiency Credential with Secondary Diploma

2P1: Postsecondary Degree or Credential

###### Core Indicator 3. Placement and Retention\*

3S1: Secondary Placement

3P1: Postsecondary Placement

3P2: Postsecondary Retention

Core Indicator 4. Participation in and Completion of Non-Traditional
Programs

4S1: Participation in Secondary Non-Traditional Programs

4S2: Completion of Secondary Non-Traditional Programs

4P1: Participation in Postsecondary Non-Traditional Programs

4P2: Completion of Postsecondary Non-Traditional Programs

\* Placement and retention address employment, further education and/or
military.

Source: OVAE Core Indicator Framework (2000)

In the OVAE core indicator framework, states are held accountable for
only those students with concentrated participation in vocational
education---that is, those students who reach a state-defined threshold
level of vocational education. Although states are required to report
enrollment information for all vocational students, they are required to
report performance information for only vocational concentrators (See
Figure 3). Only two subindicators address a wider student population.
These are the two subindicators for participation in non-traditional
programs---4S1---Participation in Secondary Non-Traditional Programs and
4P1---Participation in Postsecondary Non-Traditional Programs. These two
subindicators address vocational participants---that is, students who
enrolled in at least one vocational-technical education course. The
placement and retention subindicators further restrict state
accountability to only those vocational concentrators who successfully
complete their programs by receiving a credential. As shown in Figure 3,
vocational completers are those vocational concentrators who have
attained the academic and technical knowledge/skills/proficiencies in
their programs/sequences of courses as evidenced by a diploma or degree
or certificate.

In using the core indicator framework, states have considerable latitude
in defining their measures and measurement approaches. They also have
considerable flexibility in how they define vocational concentrators and
completers for reporting their results.

######## Figure 3: Student Population Definitions

######## 

######## Threshold Level of Vocational Education

A threshold level of vocational education is defined as a
program/sequence of courses or instructional units that provides an
individual with the academic and technical
knowledge/skills/proficiencies to prepare the individual for employment
and/or further/advanced education (Section 3 (29) Definitions).

+-----------------------------------------------------------------------+
| ###                                                                   |
|                                                                       |
| ######## Vocational Participant                                       |
|                                                                       |
| Student who enrolled in at least one vocational-technical education   |
| course.                                                               |
+-----------------------------------------------------------------------+
| ######## Vocational Concentrator                                      |
|                                                                       |
| Student who enrolled in a threshold level of vocational education.    |
+-----------------------------------------------------------------------+
| ######## Vocational Completer                                         |
|                                                                       |
| Student who attained the academic and technical                       |
| knowledge/skills/proficiencies within a program/sequence of courses   |
| or instructional units that provides an individual with the academic  |
| and technical knowledge/skills/proficiencies to prepare the           |
| individual for employment and/or further/advanced education.          |
|                                                                       |
| Source: OVAE Core Indicator Framework (2000)                          |
+-----------------------------------------------------------------------+

## Core Postsecondary Indicators: What Should We Be Measuring and How Can We Improve Data Quality? {#core-postsecondary-indicators-what-should-we-be-measuring-and-how-can-we-improve-data-quality .OVAE}

Although the development of the Perkins III core indicator framework was
successful in supporting state efforts to report performance on all core
indicators, this effort also raised some major problems and issues for
postsecondary accountability systems. This framework attempted to
develop parallel secondary and postsecondary subindicators emphasizing
consistency of measurement for schools and colleges. As found in OVAE's
state pilot project (OVAE, 2000), this effort resulted in major
challenges for states in applying some measures to postsecondary
institutions such as academic attainment. As seen in the discussion of
current trends in states in developing "core indicators of
effectiveness," this also has resulted in some major gaps in measuring
critical outcomes addressed in state and institutional systems, such as
earnings of completers and student satisfaction and accomplishment of
student goals.

The question of what we should be measuring can best be addressed by
reviewing the development of each Perkins III core indicator and
comparing the Perkins III subindicators to the state and institutional
"core indicators of effectiveness" summarized in Figure 1.

**Core Indicator 1: Student Attainment**. The effort to create parallel
measures for secondary and postsecondary education on academic
attainment poses some major problems and issues. The core indicator
framework was designed to build on and reinforce state academic
standards and assessment systems designed for all secondary students
including vocational education students. This was done to reinforce the
commitment that vocational students should be expected to meet the same
state academic requirements as all students and perform at or above the
level of all students on state academic assessments. This effort was
very timely and successful in improving secondary accountability but
presents problems at the postsecondary level. One reason is that there
is no equivalent for state academic standards at the postsecondary level
and not all programs define separate academic requirements. Another
reason is that students have varying needs and interests and may not
take courses or programs with well-define academic requirements other
than basic skill requirements.

Another problem is the lack of strong measurement approaches for
vocational and technical skill attainment. Most states utilized existing
institutional grading and credentialing systems to measure skill
attainment with the argument that they are valid and reliable indicators
of skill attainment. Many states used the same measurement
approach---program/degree completion---for both skill attainment and
Core Indicator 2---Credential Attainment---with the argument that these
two indicators were measuring the same student outcome (OVAE, 2000b).
These arguments are reasonable but may not stand the test of time in
light of the growing distrust of postsecondary assessment and
credentialing practices and growing calls for reform in assessment and
credentialing at all levels. Also, most students do not enroll in degree
and certificate programs but take courses to upgrade skills and transfer
to other institutions. The separate skill attainment measure has the
potential to capture outcomes for these students.

[Recommendations]{.underline}. Future efforts should reconsider the use
of a separate academic attainment measure. These efforts should focus on
improving the measurement of vocational and technical skill attainment
by improving the specification of learning outcomes and the assessment
and credendialing of these skills by community and technical colleges.

# **Core Indicator 2: Credential Attainment**. This is probably the most widely accepted and least controversial core indicator for postsecondary education. This is because it reinforces and supports the widespread collection of this information to meet a variety of federal and state accountability requirements. However, state measurement approaches raise some major problems and issues.

The first issue is what degree and certificate programs should be
included. As reported in the OVAE state pilot project (OVAE, 2000b),
some states include all AA/AS degree programs and all certificate
programs. Others only include AS degree programs. This restricted
interpretation fails to account for the fastest growing credentialing
market for two-year colleges---short-term certificates.

The second issue is what students should be included. Some states
include all students that completed a threshold level of courses within
a program and/or registered the intent to complete the program. Other
states use approaches from other federal reporting requirements that
only include a small subset of students who were first-time, full-time
students. This substantially reduces the number of students reported and
fails to capture the growing non-traditional student population at
two-year colleges---adults who attend part-time and work on their
programs in irregular patterns. As recommended by Alred et al. (1999),
the measurement of credential attainment should represent a broader
student population and then be reported by different student populations
as necessary.

[Recommendations]{.underline}. Future efforts should focus on gaining
broad state consensus on including the full range of degree and
certificate programs and including all students who reach a threshold
level regardless of whether they are first-time or returning students or
whether they are full-time or part-time. As discussed above, future
efforts also should focus on improving the specification and assessment
of learning outcomes that are the basis for awarding degrees and
certificates.

**Core Indicator 3: Placement and Retention**. The Perkins III framework
restricts the focus of placement and retention to measuring only the
transition of students to further education, employment, and/or the
military. The only exception is employment, which includes a measure of
employment retention. The subindicators do not address whether students
are successful at four-year institutions. And, they did not address the
quality of employment as measured by wages and earnings.

As discussed earlier, many state and institutional core indicator
systems not only address transition to four-year colleges but also
measure the success of students after transition. They use a variety of
measures including persistence, grade point averages compared to now
transfer students, and the percent of credits successfully transferred.
Many states also address the quality of employment including
training-related employment and earnings of graduates. The measurement
of employment retention and earnings is a critical linkage to workforce
development including connections to the WIA performance measures.

Another major issue is the different capacities of states to actually
collect valid and reliable data on student transitions. Some states have
developed the capacity to measure student transitions from community and
technical colleges to four-year colleges and employment within and
outside the state. However, these state capabilities vary tremendously
by state. The result is wide variations in state performance due mainly
to wide variations in data quality, not program quality. States need
further support at the federal level to improve these systems (National
Postsecondary Education Cooperative, 2001).

[Recommendations]{.underline}. Future efforts should expand the
measurement of transition to include student retention and success at
four-year colleges. Future efforts also should develop consistent
measures with workforce development to address employment retention and
earnings. Finally, future efforts should focus on improving state
student transition data systems to improve the data quality for
measuring transition to further education and employment.

# **Core Indicator 4: Non-Traditional Participation and Completion.** This core indicator is the only indicator focused on specific programs and a specific student population in vocational education---non-traditional students. This population is also a special population with states required to report performance on this population for all other core indicators.

This core indicator raises some major questions and issues. The first
issue is the wide variation in how states define non-traditional
programs using their own systems for defining "programs." (OVAE, 2000b)
The second issue is whether postsecondary education should focus on
participation in one course for 4P1 or focus more on actual enrollment
in a non-traditional program, which requires more concentration, usually
9-12 credit hours.

However, the biggest question is the need for this core indicator and
whether it should play such a large role in the performance
accountability system. Very few critics would argue that the Perkins III
system should not focus on improving the access of males and females to
non-traditional programs that result in improved success in the labor
market. The question is: what is the overall outcome we want and what
should we measure? Should the focus be on one strategy to improve equity
in opportunity and earnings or should the focus be on measuring whether
males and females are getting similar results? If so, then states and
institutions should be held accountable for equity in results and should
use non-traditional careers as one strategy in closing the employment
and earning gaps of males and females.

[Recommendations]{.underline}. Future efforts should focus on reporting
the performance of demographic groups and special populations on getting
equitable results in skill attainment, credential attainment, and
further education and employment transitions. One major special
population should be students (males and females) enrolling in
non-traditional programs. Future efforts should then review the need to
retain this core indicator as a separate stand-alone indicator for
postsecondary education.

**Additional Core Indicators: Student Goal Attainment and Customer
Satisfaction**. In comparing the Perkins III core indicator framework
with state and institutional efforts to develop their own core indicator
frameworks, there is one major difference---the measurement of
individual student goal attainment and student and employer
satisfaction. Given the diverse needs and requirements of students
attending community and technical colleges and the growth of enrollment
in short-term, individualized education and training, this requires
further exploration. In addition, the new emphasis on customer
satisfaction in higher education and government also suggests the need
for exploring the use of customer satisfaction to measure both student
goal attainment and overall satisfaction with services.

[Recommendations]{.underline}. Future efforts also should explore the
use of customer satisfaction indicators to address the attainment of
student goals and the satisfaction of students and employers with
postsecondary vocational education services.

## How Do We Reduce Burden and Improve Value? {#how-do-we-reduce-burden-and-improve-value .OVAE}

The second question is how to reduce the burden and improve the value of
the Perkins III accountability system for states and institutions?

As described earlier, state community and technical college boards and
colleges themselves are facing a new accountability environment at the
national, state and local levels. Performance reporting is now
considered a normal cost of doing business. The question of burden is
not one of simply being required to be accountable for results and be
expected to report high quality performance data to external
stakeholders and funders. The question of burden is raised in the
current environment only when reporting requirements from different
funders and stakeholders:

(1) Are not valid indicators and steer states and institutions away from
    performance measures that matter,

(2) Require too much additional requirements for reporting detailed
    performance information by multiple student populations,

(3) Are inconsistent with the reporting requirements from other funders
    and stakeholders requiring multiple data collection and reporting
    systems,

(4) Do not provide sufficient funding or support to justify and offset
    the additional costs of doing business, and

(5) Do not add value in improving the effectiveness of the organization
    in meeting stakeholder and customer requirements.

**Perkins III Core Indicators: Valid Indicators for Community and
Technical Colleges?** The question is not whether there are too many or
too few Perkins measures. The question is whether they are the right
measures and address the most critical goals of community and technical
colleges in vocational education. Do the required measures provide a
complete (scope) and accurate picture of the performance of colleges in
vocational education? Are they the measures that really matter?

As discussed earlier, most of the Perkins III core indicators match core
indicators that states and institutions currently define and measure in
their own performance indicator systems because they are the same
indicators identified by institutions themselves and requested by other
federal and state funders and stakeholders. As a result, they do not
create unnecessary burden on states and institutions. However, as
recommended above, other Perkins III measures should be reconsidered
and, in some cases, eliminated. This will not necessarily reduce the
number of core indicators because, as recommended above, future efforts
should also consider additional measures including transition success,
earnings, and student goal attainment and customer satisfaction.

**Additional Reporting Requirements: Need for Demographic Group and
Special Populations Performance Data**. Perkins III places requirements
on states to report performance by demographic group and special
population. As discussed earlier, many federal and state accountability
systems require similar reporting and there is major interest in
postsecondary education on equity in results such as the participation
and success of minorities in postsecondary education. Future efforts
should focus on developing standard definitions and measures of student
characteristics to reduce burden.

Consistency with Other Federal and State Reporting Requirements: Need
for Aligned Postsecondary Education and Workforce Development Measures.
As stated above, the real burden in the new accountability environment
is not simply in the reporting of outcomes, but in the need to establish
separate reporting systems for multiple federal agencies and state
boards. As discussed above, one major focus in future efforts should be
to increase the consistency of performance indicators and data
collection requirements across federal agencies and states.

**Reporting Requirements Backed by Funding/Technical Support: The Need
for Building Capacity in Assessment and Student Transition Data
Systems**. Another major focus in future efforts should be providing
states and institutions with the resources and technical support to
gather and report high-quality data on the most critical core
indicators. As discussed earlier, the most critical data quality
problems are in skill attainment and transition measurement. Future
efforts should focus on improving assessment and credentialing systems
at postsecondary education institutions and improving state student
transition data systems.

**Adding Value: Moving from Performance Reporting to Performance
Management for Continuous Improvement**. Future efforts should not only
reduce costs but also improve value by improving state and institutional
capacity to use data to improve performance. These efforts should focus
on promoting best practices and models for managing improvement. They
also should establish a national benchmarking system that provides
states and institutions with comparative performance information.

## How Do We Improve the Consistency and Integration of Performance Indicators? {#how-do-we-improve-the-consistency-and-integration-of-performance-indicators .OVAE}

The third question is how to improve the consistency and integration of
performance indicators?

In the new accountability environment, states and institutions are
facing performance requirements from all sides. Performance measurement
and reporting is now considered a normal cost of doing business in the
education arena. As argued above, the real burden is having to address
inconsistent or contradictory performance goals and having to develop
and maintain multiple reporting systems for multiple federal, state, and
local stakeholders. Although this problem is a general problem faced by
all postsecondary education, it has the most severe impacts on community
and technical colleges. This is because of their multiple missions and
diverse customer goals and the difficult challenges of integrating and
managing their multiple functions with different external stakeholders
and performance requirements. This also is because community and
technical colleges span two worlds---postsecondary education and
workforce development. They are the major providers of education and
training services for state and local workforce development systems
funded by the Workforce Investment Act (WIA) and must balance and
integrate both postsecondary and workforce development performance
measurement and reporting requirements.

**Problems and Issues in Previous National Efforts**. The call for
consistent performance measures is nothing new. The need for common or
consistent performance measures in postsecondary education and workforce
development has been widely recognized for almost twenty years. Federal
agencies and states have worked together over many years to align
federal performance measures and reporting requirements (Baj, Sheets,
and Trott, 1994).

The first problem faced by previous efforts was the lack of educational
performance measures. These efforts were hampered by the lack of a
well-defined set of performance measures for postsecondary education,
including vocational education, which could be integrated with workforce
development measures. This is no longer the case. The problem now is too
many different sets of postsecondary educational measures coming from
many different units and programs in the U.S. Department of Education
and other federal agencies.

A second problem with previous efforts has been their federal-centered,
systems-building approach. Previous efforts have focused on aligning
existing federal measures with only limited attention to state and local
performance requirements. This is because previous efforts were driven
by the workforce development side. They were focused on developing
comprehensive workforce development systems. Community and technical
colleges are funded primarily at the state and local levels and are
increasingly driven by state and local performance requirements that
address all of their missions including academic transfer, adult
education, and economic development. They also are expected to address
the needs and requirements of their regional economies and communities.
Future efforts to integrate measures should balance federal, state, and
local performance requirements.

A third problem with previous efforts has been the lack of attention to
how measures must be designed to promote improved performance across
functional areas at the service provider level---in this case, at the
institutional level. The assumption has been that if we develop
consistent measures across funding streams for each separate measurement
area---placement, skill attainment, program completion\-\--then we have
solved the problem. However, leading efforts in the private sector go a
step farther (e.g., Kaplan and Norton, 1996, Eckes, 2001). They attempt
to develop performance measurement systems that promote the integration
and synergy of different functional areas and units by looking at the
interrelationship between measures and whether they are having
unintended negative consequences for the entire organization. They also
develop strategic or balanced scorecard systems that can help the entire
organization manage improvement strategies and manage whole end-to-end
processes that provide value to customers. These systems help manage the
performance of the entire enterprise from the perspective of the
customer at the point of service delivery.

Moving from Federal-centered to Institution-centered Frameworks. Future
efforts should build from previous federal-centered efforts, but they
should create more balance between federal, state and local performance
requirements. They also should develop new performance indicator
frameworks from multiple perspectives, not simply the perspective of
federal agencies managing their own performance accountability and
reporting systems. They should balance this federal perspective with
state and local perspectives. However, the major focus should be the
perspective of service providers. Future efforts should develop common
or consistent measures, in part, from a comprehensive postsecondary
education framework based on the perspective of postsecondary
educational institutions and how they are expected to use these measures
to manage performance improvement. This framework should then be used by
federal and state governments to develop their performance measurement
requirements. This will result in an institution-centered framework that
will create more value and add less burden for community and technical
colleges. It will also have the best chance of being fully implemented
and achieving returns on federal and state investment.

## Conclusions and Recommendations {#conclusions-and-recommendations .OVAE}

As discussed in this paper, there are growing national and state demands
for accountability in postsecondary education. This new environment is
prompting federal agencies, state boards and agencies, and institutions
to develop new performance indicator systems for reporting and managing
continuous improvement for all major functions and programs including
vocational education. What role should the U.S. Department of Education
play in the future and how should the department build from the
accomplishments of the Perkins III effort?

## Federal Role in Postsecondary Accountability  {#federal-role-in-postsecondary-accountability .OVAE}

The federal government is a major funder and stakeholder in
postsecondary vocational education through Perkins funding and other
federal funding sources (e.g., student grants and loans) that are
increasingly used by students attending community and technical
colleges. The federal government has a legitimate role to play in
ensuring that states and institutions are meeting the needs of targeted
customers and providing a return on federal investment.

However, the federal government has unnecessarily increased state and
local accountability burdens by not developing consistent performance
indicators across multiple federal programs and working with states and
institutions to develop a common performance indicator framework that
can be used to promote improvement in postsecondary education including
vocational education. There is a need to integrate the Perkins III
secondary core indicator framework within state secondary educational
accountability systems. And, there is also a need to do the same thing
at the postsecondary level by integrating future efforts into larger
efforts to develop a core indicator framework for postsecondary
education. This effort should place a strong emphasis on integrating
postsecondary education and workforce development.

To address this problem, the federal government should play a catalyst
role in working with states and institutions to develop a national (not
federal) consensus on an institution-centered core indicator framework
for two-year colleges that address the full range of goals and functions
of these unique institutions and address the needs of all funders and
stakeholder, not just the federal government. This framework should
address both performance indicators and student populations. The federal
government should then use this framework to develop federal performance
indicators and measures and data reporting requirements consistent with
this framework.

The federal government also should play a major role in expanding the
capacity of states and educational institutions to collect and use data
to report performance and manage continuous improvement. The federal
government should focus on improving assessment and credentialing
practices and improving the capacity of states and institutions to
measure student transitions. Also, the federal government should work
with states and institutions to improve data management and analysis
capacity at the state and local levels.

The federal government also should play a catalyst role in shifting the
focus from reporting performance to using performance information to
manage continuous improvement. This should be done through two
strategies. First, the federal government should work with states and
institutions to develop data and technical standards for information
systems and promote best practices in state and local information
systems. This should provide the basis for improved federal reporting.
Second, the federal government should promote the use of data for
performance improvement by promoting best practices in performance
management and continuous improvement and encouraging the development of
a national benchmarking data system for states and institutions to use
in obtaining comparative performance information.

## Building from the Perkins III Effort {#building-from-the-perkins-iii-effort .OVAE}

As discussed earlier, the development of the Perkins III accountability
system for postsecondary vocational education broke new ground and
improved the measurement and reporting of core secondary and
postsecondary student outcomes in vocational education programs. It also
used core indicators that are widely used by states and institutions in
developing core indicators of effectiveness.

Future efforts should build on the Perkins III accomplishments by
integrating major elements of this framework into a larger
institution-centered postsecondary framework. This effort should address
some of the major problems including: (1) data quality problems in skill
attainment measurement, (2) including all students and programs in
credential attainment measurement, (3) expanding transition measurement
to include success in four-year educational programs and earnings and
improving state student transition data systems, and (4) recasting the
non-traditional performance measure. These efforts also should explore
adding additional measures to capture the achievement of student goals
and customer satisfaction.

## Major Recommendations {#major-recommendations .OVAE}

1.  OVAE and states should establish a national taskforce to develop a
    comprehensive institution-centered core indicator framework that
    two-year community and technical colleges can use to manage
    continuous improvement, provide consumer information, and report
    progress to external stakeholders through federal and state
    performance accountability systems. This framework minimally should
    establish a core set of performance indicators and student
    populations.

2.  This national taskforce should focus attention on the development of
    core indicators in two major areas: (1) skill and credential
    attainment and (2) transition to further education and employment.
    It should explore the development of new transition measures that
    address the success and retention of students transferring to
    four-year colleges and universities. It also should explore new
    transition measures that address employment retention and earnings
    consistent with related measures in workforce development.

3.  This national taskforce should sponsor a national roundtable
    discussion on best practices in the measurement and use of customer
    satisfaction data in the public and private sectors with a special
    focus on how corporations use customer satisfaction data in their
    performance management systems. The national taskforce should use
    this forum to consider additional core indicators for customer
    satisfaction that can address student goal attainment and overall
    student and employer satisfaction with services.

4.  This taskforce should also focus on developing common definitions
    for all major student populations for reporting performance on the
    core performance indicators. The taskforce should focus on defining
    the major demographic groups and special populations.

5.  OVAE and its federal agency partners and states and national
    associations should use this core indicator framework to promote
    consistency in performance measurement requirements in federal and
    state legislation and consistency between the performance indicators
    developed and used by federal and state agencies and programs.

6.  OVAE and states should focus future data quality efforts on two
    major areas: (1) assessment and credentialing systems used by
    postsecondary institutions to award degrees and certificates and (2)
    transition measurement especially the capacity of states to use
    administrative record exchange systems to measure student
    transitions to four-year colleges and universities and UI-covered
    employment both within and between states.

7.  OVAE and states should identify, develop, and promote models for
    institution-based performance management systems for managing
    continuous improvement.

8.  OVAE and states along with representatives from community technical
    colleges should launch a national project to use this
    institution-based performance indicator framework and leading
    performance management models as the foundation for establishing a
    comprehensive set of national (not federal) voluntary data content
    and technical standards for state and institutional information
    systems for the collection and distribution of information.

9.  OVAE and states should use these voluntary national data content and
    technical standards to develop federal and state reporting
    requirements and to develop a national benchmarking data system that
    states and institutions can use to benchmark their performance and
    manage their improvement efforts.

## References {#references .OVAE}

Alred, Richard (1997), "From Closed to Open Systems: New Designs for
Effectiveness in Community Colleges," [Journal of Applied Research in
the Community Colleges]{.underline}, 5(1): 9-15.

Alred, Richard, Peter Ewell, James Hudgins (1994), [Core Indicators of
Effectiveness for Community Colleges]{.underline}, Washington, D.C.:
American Association of Community Colleges, Community College Press.

Ashworth, Kenneth H. (1994), "Performance-Based Funding in Higher
Education: The Texas Case Study," [Change]{.underline},
November/December 1994, pp. 9-15.

Baj, John, Robert G. Sheets, and Charles E. Trott (1994), [Building
State Workforce Development Systems Based on Policy Coordination and
Quality Asssurance]{.underline}, Washington, D.C.: National Governors'
Association.

Bogue, E. Grady and Robert L. Saunders (1992), [The Evidence for
Quality: Strengthening the Tests of Academic and Administrative
Effectiveness]{.underline}, San Francisco, CA: Jossey-Bass Publishers.

Borden, Victor M.H. and Karen V. Bottrill (1994), "Performance
Indicators: History, Definitions, and Methods," in Victor M.H. Borden
and Trudy W. Banta (eds.), [Using Performance Indicators to Guide
Strategic Decision Making]{.underline}, New Directions for Institutional
Research, San Francisco, CA: Jossey-Bass Publishers, pp. 5-22.

Eckes, George (2001), [The Six Sigma]{.underline} Revolution, New York:
John Wiley and Sons.

Erwin, T. Dary (1999), "The United States Perspective on and Experience
of Performance Indicators and Threshold Standards---How Is Quality
Determined?"

In Helen Smith, Michael Armstrong, and Sally Brown (eds.), [Benchmarking
and Threshold Standards in Higher Education]{.underline}, London: Kogan
Page.

Ewell, Peter (1994), "A Matter of Integrity: Accountability and the
Future of Self-Regulation," [Change]{.underline}, November/December
1994, pp. 25-29.

Kaplan, Robert S. and David P. Norton (1996), [The Balanced Scorecard:
Translating Strategy into Action]{.underline}, Boston, MA: Harvard
Business School Press.

Kulik, James A. (2001), "Student Ratings: Validity, Utility, and
Controversy," in Michael Theall, Philip C. Abrami, Lisa A. Mets (eds.),
[The Student Ratings Debate: Are They Valid? How Can We Best Use
Them?,]{.underline} New Directions for Institutional Research, San
Francisco, CA: Jossey-Bass Publishers, pp. 27-44.

Levine, Arthur (1992), "Why Colleges Are Continuing to Lose the Public
Trust," [Change]{.underline}, July/August, 1992, p. 4.

Liner, Blaine, Harry P. Hatry, Elisa Vinson, Ryan Allen, Pat Dusenbury,
Scott Bryant, Ron Snell (2001), [Making Results-Based State Government
Work]{.underline}, Washington, D.C.: The Urban Institute.

Mundhenk, Robert T. (2000), "Institutional Effectiveness and
Unemployment Insurance Data," White Paper of the American Association of
Community Colleges.

National Alliance of Business (2001), [Six Principles of Certificate
Quality: The Employer Perspectives]{.underline}, Washington, D.C.:
National Alliance of Business.

National Center for Public Policy and Higher Education (2000),
[Measuring Up 2000: The State-By-State Report Card for Higher
Education]{.underline}, Washington: D.C.: National Center for Public
Policy and Higher Education.

National Governors' Association (2000), [Higher Expectations: Essays on
the Future of Postsecondary Education]{.underline}, Washington, D.C.:
National Governors' Association.

National Postsecondary Education Cooperative (2002), [Student Transition
Data Systems]{.underline}, Report of the Working Committee.

Office of Vocational and Adult Education, U.S. Department of Education
(2000a), [Perkins III Core Indicator Framework]{.underline}.

Office of Vocational and Adult Education, U.S. Department of Education
(2000b), [Postsecondary Pilot Project Report]{.underline}.

Seppanen, Loretta J. (1995), "Linkages to the World of Employment," in
Peter T. Ewell (ed), [Student Tracking: New Techniques, New Demands, New
Directions for Institutional Research]{.underline}, San Francisco, CA:
Jossey-Bass Publishers, pp. 77-91).