NATIONAL CENTER FOR EDUCATION STATISTICS

Research and Development Report November 1996

{

‘!

The National Assessment of

College Student Learning:

An Inventory of

I

,

1

\’

State-Level Assessment

Activities

A Report of the Proceedings of the Third

Study Design Workshop

Sal Corrallo, Project Officer

Educational Assessment Division

National Center for Education Statistics

U.S. Department of Education

Office of Educational Research and Improvement NCES 96-862

For sale by the U.S. Government Printing ~ce

Suraimendent of Documents Mail Stop SSOP Washington DC 20402 9328
U.S. Department of Education

Richard W. Riley

Secretary

Office of Educational Research and Improvement

Sharon P. Robinson

Assistant Secretary

National Center for Education Statistics

Pascal D. Forgione, Jr.

Commissioner

The National Center for Education Statistics (NCES) is the primary federal entity for collecting,

analyzing, and reporting data related to education in the United States and other nations. It

fulfills a congressional mandate to collect, collate, analyze, and report full and complete

statistics on the condition of education in the United States; conduct and publish reports and

specialized analyses of the meaning and significance of such statistics; assist state and local

education agencies in improving their statistical systems; and review and report on education

activities in foreign countries.

NCES activities are designed to address high priority education data needs; provide consistent, reliable, complete, and accurate indicators of educatiofi status and trends; and report

timely, useful, and high quality data to the U.S. Department of Education, the Congress, the

states, other education policymakers, practitioners, data users, and the general public.

We strive to make our products available in a variety of formats and in language that is

appropriate to a variety of audiences. You, as our customer, are the best judge of our success

in communicating information effectively. If you have any comments or suggestions about

this or any other NCES product or report, we would like to hear from you. Please direct your

comments to:

National Center for Education Statistics

Office of Educational Research and Improvement

U.S. Department of Education

555 New Jersey Avenue NW

Washington, DC 20208-5574

November 1996

Contact:

Education Assessment Group

(202) 219-1761

The NCES World Wide Web Home Page is

http:/Avww.ed.gov/NCESl

Suggested Citation

U.S. Department of Education. National Center for Education Statistics. The IVatjonal

Assessment of Co//ege Student Learning: An /nventory of State-Leve/Assessrnent Activities,

A Report of the Proceedings of the Third Study Design Workshop, NCES 96-862, by Sal

Corrallo, project director, Washington, DC: 1996.
-.

CON TENT S

Page

1, Background a]ld Over\, iew . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ...1

A. Bacl<grollnd . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ...1

B. orga]lizationoftlle\ Vor]<s]lop . . . . . . . . . . . . . . . . . . . . . . . . . . ...2

C. OrganizationoftlleProceedings . . . . . . . . . . . . . . . . . . . . . . . . . ...3

11. Inventory of State Assessment Activities . . . . . . . . . . . . .

. . . . . . . . . .

. . 3

A. The Current Status of State Assessment Activity . . . . . . . ‘. . . . . . . . . . . .5

B. Perceived Obstacles and Needs... . . . . . . . . . . . . . . . . . . . . . . ..10

III. Discussion Thenles and Conclusions . . . . . . . . . , . . . . . . . . . . . . . . . . 12

A, Thepolitica] Context . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ...12

B. The Methodological Context . . . . . . . . . . . . . . . . . .

C. Resu]ts of :“” ””.. . . . . 14 JVorking Sessions . . . . . . . . . . . . . . . . . . . . . . . . . . ...15

IV. Concltlsio]ls and Next Steps . . . . . . . . . . . .

. . . . . . . . . .

. . . . . . . . 19

A. Some Policjr Implications . . . . . . . . . . . . . . . . . . . . . . . . . . . ...19

B. Some Possible Next Steps . . . . . . . . . . . . . . . . . . . . . . . . . . . .. ,21

References . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . .

. . . . . ... + 24

Appendix A-1 Workshop Agenda . . . . . . . . . . ., . . . . . . . . . . . . . . ...25

Appendix A-2 Participant List . . . . . . . . . . . . . . . . . . . . . . . . . . . . ...29

Appendix A-3 NCES Background Paper . . . . . . . . . . . . . . . . . . . . . . ...39

Appendix B State Background Papers . . . . . . . . . . . .

. . . . . .. @#. .

. . . 43

.4ppendix C Texts of Prepared Conference Papers . . . . . . . . . . . . . . . . . 119

. . .

FOREWORD

America’s workers are facing their biggest challenge in a generation. They must have the skills

and knowledge, and the flexibility to use them, to help American firms create goods and services

that can compete with those from other nations. In response to this need, the nation’s governors

in 1989, and later the U.S. Congress though the Goals 2000 legislation, outlined a set of National Education Goals. The ultimate objective of the National Education Goals was enhancement of learning in this nation. Goal 6, Objective 5, focused upon the higher order thinking and

communication skills needed by college graduates, in the workplace and for the practice of citizenship.

The National Center for Education Statistics (NCES) has been concerned with the assessment of

student learning in the nation’s schools and colleges. Over the past 30 years, it has assessed student learning at the K-12 level. In preparation for the assessment of student learning at the college level, NCES has conducted three workshops over the past four years. The first identified

issues and concerns related to assessment of college student learning, the second was intended to

identify more clearly the skills and sub-skills needed, while the third was focused upon state assessment activities in 1996 with special emphasis on the skills cited in the National Education

Goals. This publication reports on the results of the third workshop.

The workshop was held in Arlington, Virginia, on December 7–8, 1995. It was conducted in cooperation with the Education Commission of the States, the State Higher Education Executive

Officers Organization, and the National Center for Postsecondary Teaching, Learning, and Assessment. Workshop proceedings were compiled by Peter Ewell of the National Center for

Higher Education Management Systems. The workshop conference contractor was Professional

and Scientific Associates of McLean, Virginia, ably represented by Regina Guyther. Sal Corrallo, Project Director for the National Assessment of College Student Learning, was conference

coordinator. Our thanks go to all who attended and contributed, especially the state participants

who prepared the background papers and other information.

Pascal D. Forgione, Jr.

Commissioner, NCES
>

THE NATIONAL ASSESSMENT OF COLLEGE STUDENT LEARNING:

AN INVENTORY OF STATE-LEVEL ASSESSMENT ACTIVITIES

A REPORT OF THE PROCEEDINGS OF THE

THIRD STUDY DESIGN WORKSHOP

I. Background and Overview

This document summarizes proceedings and conclusions of a two-day national planning workshop on the topic of furthering the assessment of national postsecondary outcomes, sponsored by

the National Center for Education Statistics (NCES) and held in Washington, D.C. on December

7-8, 1995. Activities at the workshop were designed specifically to address the degree to which

state-level assessment initiatives in higher education might aid in the construction of a national indicator of postsecondary attainment consistent with Goal 6.5 of the National Education Goals,

and to determine ways in which NCES and the states might work more effectively to develop mutuall y-supporting activities and policies in the realm of postsecondary assessment. As a consequence, invited participants consisted of representatives from each state, selected assessment and

national policy experts, and NCES staff. Conclusions of the workshop indicate that there is no immediate possibility of aggregating existing state-level data on Tostsecondary outcomes to create a

usable national indicator. But they do suggest a number of other ways in which the states and the

federal government might work together to improve the quality of data available on this topic.

A. Background

The topic of assessing the outcomes of postsecondary education emerged as an area of considerable policy concern in the early 1980’s. Like its counterpart in K-12 education, the

topic was fueled by concerns about performance, a growing need for accountability related

to increasing levels of public investment and above all, an interest in stimulating higher levels of quality. A powerful potential impetus for both conversations was the National Education Goals agreed upon by former President Bush and the nation’s governors in 1989.

Though chiefly concerned with the improvement of elementary and secondary education,

outcomes of postsecondary education were explicitly addressed by the Goals. In particular,

Goal 6, Objective 5 of both the original document and its current embodiment in President

Clinton’s “Goals 2000” requires that “the proportion of college graduates who demonstrate

an advanced ability to think critically, communicate effectively, and solve problems will increase substantially. ” Goal 3, Objective 2 also contains references to communications and

thinking skills, along with several other subjects, at both the K-12 and postsecondary levels.

In the four years subsequent to the establishment of the National Education Goals, NCES

conducted two planning workshops and funded a number of studies intended to develop a

means to assess postsecondary learning consistent with Goal 6.5 (see Appendix A-3). The

first study design workshop, held in 1991, was designed to identify the specific issues and

concerns that might surround such an assessment, and involved a wide cross-section of domain and assessment expertise. A second, held in 1992, began the process of specifying the

required assessment domain by identifying specific skills, knowledge, and abilities associated with the attainment of the abilities noted under Goal 6.5. Results of the workshops led

1
. .

.

to the development of a formal Request for Proposal (RFP) that focused on further skillsspecification suitable in detail and form to guide the development of a national survey of

collegiate learning. That RFP, however, was later withdrawn due to lack of funds.

This action, however, has not diminished the demand for information from policy makers,

employers, and the general public about how well college graduates are prepared to exercise

workplace skills and to fulfill the obligations of citizenship. To help meet this demand,

NCES attempted to identify alternative approaches to gathering information on national collegiate attainment. To that end, a number of more limited studies were sponsored. These included a Delphi study by the National Center for Postsecondary Teaching Learning and

Assessment (NCTLA) at Penn State designed to provide an initial listing of the domain, a

National Job Analysis study intended to define a generic set of job skills in high-performance workplace settings, and completion of the National Adult Literacy Study (NALS) in

1994 which contained a restricted non-systematic sample of college graduates of all ages.

Given the limits of these activities in actually generating data on collegiate attainment,

NCES decided to turn to the states for assistance. By 1989, it was known that some twothirds of the states had developed assessment policies governing their public higher education systems (Ewell, Finney and Lenth 1990). No systematic analysis of state-level

assessment activities in postsecondary assessment had been conducted since 1989, however.

NCES was therefore interested in determining botl-i the c%rrent extent and character of such

activities and, more particularly, the degree to which their results might be able to provide

an insight into the progress being made nationally toward the achievement of Goal 6.5. Because it was expected that states would differ in both capacity and approach, the posed question was not whether raw state results could be summarized into a single indicator. Rather it

was the degree to which these many state efforts might help to paint a broader collective picture of collegiate attainment in relation to broadly-identified workplace and societal skills.

B. Organization of the Workshop

The workshop brought together representatives responsible for postsecondary assessment

activities in the fifty states and selected territories, with assessment experts and selected

NCES staff (see the participants list included as Appendix A-2). Its design included both a

set of pre-workshop inventorying activities and a working agenda intended to address four

questions:

●

●

●

●

what specific areas of knowledge, skills, and attributes are being assessed at the state

level, and what is their commonality both across states and with the skills identified in

Goal 6.5?

how are these outcomes being assessed, with what frequency are they being reported, and

how are the results being used?

what might be done to help states broaden and enhance their assessment efforts in ways

that would benefit both the states and meet the need for national reporting?

what is the appropriate role of NCES in assisting states and institutions to gather and report better information about postsecondary results?

2
Pre-workshop activities were of several kinds. First, in order to provide an updated summary of current state postsecondary assessment initiatives, states were requested to prepare

background papers describing their current activities, and to bring these to the workshop

(see Section II below). Second, a number of additional background papers were commissioned or assembled for prior distribution to participants and/or for delivery at the workshop. These included a report by ACT on state and institutional assessment needs, briefcase

studies of assessment efforts and related activities in several states, and summaries of assessment instruments and approaches by researchers and assessment organizations. Selected papers in this series are provided as Appendix C of these Proceedings.

The workshop agenda itself consisted of a mix of plenary and working sessions (see Appendix A-1 ). Plenary sessions were organized around three broad topics+ ument state assessment activities and approaches, the political context for assessment, and methodological

considerations involved in postsecondary assessment. Working sessions on Thursday evening and Friday morning were intended to explore respectively the current potential of state

data-collection efforts to inform national discussions of collegiate attainment, and the kinds

of actions that might be taken at both the state and federal levels to further postsecondary assessment efforts. The workshop ended with a final plenary designed to advance and refine

some tentative conclusions.

C. Organization of the Proceedings - -

The text of these Proceedings follows the logic of the workshop itself, with supporting documents and background papers provided as Appendices. Section II presents results of the inventory of state activities conducted as part of the workshop, and is intended to update

collective understanding of current state-level postsecondary assessment activities. As such,

its publication and dissemination constitutes one important intended outcome of the meeting. Section III describes the major themes that emerged from workshop discussions—both

in the plenary and provided by the various working groups. These are discussed under three

main subheadings—the current political context for postsecondary assessment, the corresponding methodological context, and specific actions that might be taken. Section IV provides a summary of possible next steps about which consensus emerged for both the states

and NCES. Appendices to the document include background materials for the workshop

(Appendix A), the full texts of all state background papers prepared as part of the workshop

(Appendix B), and the texts of selected additional background papers prepared by participants (Appendix C).

II. Inventory of State Assessment Activities

An important prerequisite of the workshop was to determine the extent and character of current

state-level activities in assessing postsecondary student outcomes. Accordingly, each state was requested to inventory such activities in the form of a background paper to be shared at the workshop. More particularly, the papers were intended to determine the degree to which the results of

such activities are sufficiently consistent with one another and with the domains addressed by

Goal 6.5 to provide an initial basis for constructing a national indicator of collegiate achievement. In this respect, the information provided by the state background papers indicated that current state programs could not provide such a basis. States vary considerably in their approaches

3
and policies, and state assessment programs as a body do not provide a foundation for meaningful national reporting. But the papers also revealed many themes and problems in common across

states that were useful to participants in thinking about collective needs and about how federal

and state actors in the realm of assessment might work together more effectively.

The resulting summary of state assessment activities built upon two previous national inventories

of this kind. The first such fitly-state inventory was compiled by the Education Commission of

the States (Boyer et. al. 1987), and took place when states were just beginning to implement formal policies on assessing collegiate achievement. At that point, some fifteen states had established visible programs and more than half were exploring the issue—though few contemplated

deploying the kinds of large-scale assessments using standardized instruments that were then

common in K-12 education. The second such inventory was compiled three years later by the

Education Commission of the States and the State Higher Education Executive Officers (Ewell,

Finney and Lenth 1990). By this point, more than two-thirds of the states had adopted or were developing formal assessment policies, and only nine indicated that they had no plans of this kind.

Again, however, very few standardized assessment methods were mandated and most such policies required institutions to develop their own assessment approaches consistent with local missions and student clienteles.

The 1995 inventory conducted through the workshop was intended partly to update the information obtained through these prior efforts. More specifically, itwas aimed at determining areas of

potential common interest and the alignment of current state activities with the outcomes domains associated with Goal 6.5—the abilities to “think critically, communicate effectively, and

solve problems. ” To that end, the state background papers consisted of two components. First,

states were requested to describe the origins and development of their approach and, more specifically, to indicate the kinds of instruments employed and their potential for contributing information useful to a national indicator of collegiate achievement. The full texts of all state

submissions under this heading is included as Appendix B. Secondly, states were asked to identify important obstacles to furthering their assessment initiatives, important methodological problems involved in developing assessment approaches, and what they saw as the most important

needs to move a national assessment agenda forward.

In all, thirty-four states and Puerto Rico prepared background papers as requested. While the formal response was less than that achieved by the prior two inventories, a combination of followup efforts and a review of available published documents allowed an accurate basic inventory of

activity to be constructed for all fifty states and Puerto Rico. Papers were prepared largely by

SHEEO’S or their designees, and the majority attached additional documentation describing their

policies, approaches, and results obtained. Contextual information was also supplied by two additional studies conducted by other agencies. The first was a SHEEO survey of state-level management information system capabilities which yielded a positive picture of the evolution of

state-level technical capacity in the area of student information (Russell and Chisholm 1995).

The second was an interview-based study of education leaders in forty-nine state legislatures conducted by the National Education Association, which provided useful insights into the shifting

political context of accountability policy in higher education (Ruppert 1996).

4
●

A. The Current Status of State Assessment Activity

Results of the 1995 inventory suggest that little of substance has changed with respect to

state-level assessment since 1989. More states are now involved in the activity, but the additions are those which had previously indicated plans for doing so. Most states, moreover, are

continuing to support an “institution-centered” approach that emphasizes local development

and use of results and contains no common measures. But results of the 1995 inventory also

< suggest a wider political context for assessment significantly different from that apparent in

1989. Especially salient here is a growing policy preoccupation with governance and fiscal

matters at the state level, effectively displacing the agenda of undergraduate educational improvement that originally gave rise to both assessment and the National Goals movement in

the eighties.

1. Policy Patterns. Chart 1 presents the current status of the states with respect to

postsecondary assessment initiatives and describes these initiatives across a number of policy categories. About half the states are maintaining the “institution-centered” policy approach pioneered by such states as Virginia and Colorado about a

decade ago. This approach emphasizes the development of institution-level assessment methods that best fit the mission and student clientele of each individual college or university. Most are governed by state-!evel guidelines for the

development of assessment measures, but require no commonality across institutions. Most also require periodic public reporting of results by institutions either

annually or biennially. About two-thirds of such initiatives remain board-mandated and involve no explicit legislation. The remaining third are explicitly required by legislation.

While the inventory indicated no change in the dominance of this institution- centered policy approach to assessment, results did suggest a number of changes in

the ways in which states were carrying it out. Most apparent here was a growing

de-emphasis on active enforcement, largely because of policy preoccupation with

other matters. Though policies are in place, there was significant evidence that

compliance is being de-emphasized and that assessment has become a “backbumer” activity in many states. Coupled with this was growing fmstration with

the institution-centered approach as a means to effectively address growing demands for public accountability. As a result, in about a third of the states falling in

the “institution-centered” policy category, institutional reporting is being supplemented by the development of common—but largely non-outcomes-based—indicaters of performance.

5
-.

Chart 1

Status of State Assessment Initiatives in 1995

Initiative Date Type of Common Public

Name Established Initiative Instruments Reporting

AL

AK

AZ

AR

CA

co

CT

DE

FL

GA

HI

ID

IL

IN

1A

KS

KY

LA

ME

MD

MA

MI

MN

MS

MO

MT

NE

NV

NH

NJ

NM

NY

NC

ND

Assessment Policy

(not available)

Regents Mandate

Act 98

HE Assessment Act

HB 1187

Strategic Plan

(no initiative)

CLAST

Planning Policy

Acts 371

Outcomes Policy

Review of UG Educ.

(no initiative)

Program Review

Assessment Policy

KAEP

(not available)

Planning Goals

HE Reorg. Act

1990

1986

1989

1990

985

989

981

989

989

988

986-90

1991

1988

1990

1988/94

1988/9 1

(no initiative-’’voluntary” guidelines)

(no initiative-no authority)

(no initiative-no authority)

(not available)

Assessment Program

(no initiative)

(Program Review)

Assessment Policy

986/87

989

Board Goals 198

(no initiative-’’evolving situation”)

Report Card 1990

Board of Ed Policy c1980

Board Policy —

Strategic Plan 1996

Institution-Centered

Institution-Centered

Comprehensive

Institution-Centered

Institution-Centered

Institution-Centered

Gatekeeping Test

Institution-Centered

Institution-Centered

Institution-C_enter~d

Institution-Centered

Institution-Centered

Institution-Centered

Institution-Centered

Institution-Centered

Institution-Centered

Institution-Centered

Institution-Centered

Institution-Centered

Institution-Centered

Program Review

Program Review

Accreditation-Oriented

No

No

Yes

No

No

No

Yes

Yes

No

No

No

No

No

No

No

No

No

No

No

No

No

No

No

Voluntary

Annual

Annual

By Systelm

Annual

Voluntary

Annual

Annual

Annual

Periodic

Periodic

5-Yr. Cycle

Annual

Annual

Periodic

Annual

Annual

Biennial

Voluntary

Annual

Cyclical

Cyclical

Cyclical

6
Chart 1 (Continued)

Status of State Assessment Initiatives in 1995

Initiative Date Type of Common Public

Name Established Initiative Instruments Reporting

OH

OK

.? OR

PA

R]

Sc

SD

TN

TX

UT

VT

VA

WA

Wv

WI

WY

PR

SB140 1989

Regents Policy 1991

Board Policy 1991

(no initiative-no authority)

Board Policy

Cutting Edge/Act 255

Assessment Policy

Performance Funding

TASP

Assessment Policy

(no initiative)

Assessment Program

Assessment Policy

Assessment Policy

Accountability Policy

1986

1988/92

1996

1979

1989

1992

1986

1989

1989

1993

(no initiative-no authority)

Assessment Policy

Institution-Centered

Institution-Centered

Institution-Centered

Institution-Centered

Institution-Centered

Comprehensive

Comprehensive

Basic Skills/Gatekeeping

Institution-Centered

Institution-Centered

.

Institution-Centered

Institution-Centered

Comprehensive

Institution-Centered

No

No

No

No

No

Yes

Yes

Yes

No

No

No

No

Yes

No

Annual

Annual

Annual

Voluntary

Annual

Annual

Annual

Annual

Biennial

Biennial

Biennial

Periodic

3-Yr. Cycle

Periodic

For both reasons, moreover, a number of states are actively attempting to “streamline” their approaches to assessment reporting. Virginia, for instance, is adopting a

peer-based oral reporting and review process and Colorado is examining ways to

simplify required written reports.

Only about fifteen percent of the states reported that a common collegiate outcome

measure of any kind was in place or under development. This set of states, in turn,

reported using a varied set of instruments and approaches (see Chart 2). Florida,

Texas and Georgia continue to maintain competency testing programs in at least

one domain for all students in public institutions—programs that have been in

place for more than a decade. Similarly well established is Tennessee’s “performance-funding” approach, which tests samples of students in a range of general education domains as a part of a statewide incentive-allocation system. Newer

programs follow these leads. Wisconsin periodically examines samples of students

using national tests of general education—an approach also followed by South Dakota in 1986-88, while Arkansas uses similar instruments but tests all students—

an approach being adopted by South Dakota as well through a mandate j ust

enacted by its Board of Regents after an eight-year hiatus in common testing.

7
Though the instruments used in these seven states roughly address some of the domains of Goal 6.5, they are sufficiently varied in content that meaningful aggregation or comparison as part of a national indicators approach is not feasible.

A further four states included in the “institution-centered” policy category report

that they are “actively considering” common testing in order to help address growing demands for public accountability, while two more require that institutions use

at least one nationally normed examination as part of their “institution-centered”

approach. At the same time, survey results suggest that state interest in common

measures remains strong. Seventeen states now collect and report comparative

Chart 2

Instruments Employed by States with Common Testing

Test-taking

State Instrument Population

AR ACT-CAAP All Students

FL CLASTAII Students

GA Regents Writing Exam All Students

SD ACT-COMP ( 1986-88) Institutional Samples

ACT-CAAP (1 996) All Students

TN ACT/COMP/C-BASE Institutional Samples

TX TASP All Students

WI ACT-CAAP Institutional Samples

In addition: AL, CO, VA, WV “considering” common testing

indicators of institutional performance. In addition, according to the 1995 SHEEO

survey, some thirty-five states now compile and report institutional graduation/completion rates centrally using common definitions-a result that would

have been unthinkable a decade ago (Russell and Chisholm 1995). Finally, in

other parts of their background papers, states overwhelmingly cited “lack of appropriate instruments” and “costs of implementation and development” as their primary reasons for not moving forward on this agenda, while they mentioned far

fewer substantive policy objections to this approach. The absence of common outcomes measures across states thus appears less a matter of deliberate policy

choice than one of operational and fiscal necessity.

2. A Changing Context for State Assessment Policy. In many ways more significant than the overall pattern of state activities in collegiate assessment was their

changing character. The state background papers suggested strongly that the policy premises for adopting assessment had shifted markedly since the topic’s emergence as a notable realm of state action in the mid-eighties. First, states indicated

that assessment is becoming part of a much larger state-level policy picture. While

observers of emerging state assessment initiatives in the mid-eighties characterized many as “trains on their own track” not visibly connected to wider net8
;-

works of regulation and incentive (Ewell and Boyer 1988), most of the assessment

policies reported in the 1995 inventory were widely linked to other initiatives.

One dimension of this deficiency already mentioned is the continuing development of statewide performance indicators that include more than just outcomes.

Confirming the results of a 1993 study conducted by the Education Commission

of the States (Ruppert 1994), information obtained from some seventeen states

suggested that assessment results are being incorporated into broader accountability reports that also include information like completion rates, faculty workloads, student experiences, and instructional costs. Also more apparent are visible

linkages to funding. While in 1989 only one state program (Tennessee) tied an institution’s performance on student achievement measures with additional funding,

the 1995 inventory documented three more such cases (Missouri, Kentucky and

Arkansas), with about ten further states operating addition-to-base or categorical

finding programs with some connection to outcomes.

More fundamentally, states reported that assessment activities are embedded in

the architecture of a number of new or “re-structured” approaches to basic operations. Probably the most prominent example here is new ways to address collegiate admissions, with several states describing attempts to replace traditional

college entrance examinations with competency-based approaches founded on

portfolio review or performance-based assessment. The most visible such initiatives are in Oregon, Kentucky, and Florida. Other states noted that actions such as

these were under consideration and that ongoing K-12 reforms and the national

“New Standards” project in elementary/secondary education is increasing the political pressure on higher education to demonstrate its own effectiveness. Similar

developments were documented in the realm of inter-institutional articulation in

other states, where traditional course-by-course approaches were becoming increasingly burdensome. Finally, a number of western states—led by Utah and

Colorado-are exploring the establishment of a technology-based “virtual university” to credential achievement, using authentic assessment approaches. All three

types of initiatives build assessment activities directly into their foundations, but

for operational not indicative purposes.

The fact that such alternatives are being considered at all, moreover, is a product

of significantly altered political conditions. As noted, most states suggested that

assessment as a policy agenda had moved to a “back-burner” largely because of

the need to attend to much more basic problems. The first of these is sheer political instability. Indeed, two of the agencies reporting on their assessment activities

in 1989 had been abolished by 1995—including one of the most prominent and

comprehensive such programs (New Jersey). Several others noted that disproportionate levels of agency attention were being devoted to matters of sheer bureaucratic survival. A second such problem is fiscal. While state higher education

budgets are generally better than two years ago, many states indicated that assessment could no longer be justified as an “add-on” expenditure intended solely to

provide data, but instead had to be made part of a set of wider re-structuring initiatives intended to improve systemic efficiency. In this regard, a number of states

explicitly mentioned the fact that regional accreditors were beginning to assume

9
the major burden of stimulating “institution-centered” assessment activities, and

that independent state action in this arena may no longer be needed. Finally, most

agreed that accountability demands on higher education are much more urgent

and sharply-focused than a decade ago and, as noted, that decentralized assessment approaches are inadequate to meet such demands. These sentiments appeared to confirm results of an ACT survey on the future demand for assessment

whose results were also reported at the workshop (Steele and Lutz 1995): if credible instruments and adequate funding were made available, states would probably

have little hesitation in adopting broader assessment programs. In the short run,

however, few states appeared confident that either precondition could be met.

Reflecting this final point was considerable uncertainty among states about the degree to which the improvement of higher education quality itself remains a major

public policy issue. As noted, this was the issue ‘that originally spawned assessment in the early eighties, and the rhetoric of quality has been strongly linked to

assessment ever since. Consistent with results of the NEA study of legislative leaders (Ruppert 1996), however, state responses suggested that quality improvement

in higher education was less a leading item on most state political agendas than it

had been in the past. On the one hand, issues of health care, prisons, and tax reform are crowding out issues of educationgenegally. On the other, political leaders are increasingly seeing higher education as a “private good” that principally

benefits individuals. As a result, they are far less willing to use scarce public

funds to enhance it. Both developments, if true, have profound implications for assessment as an element of policy. The first suggests that assessment will only command a political constituency if it is linked to more fundamental changes in the

way higher education operates. Funding a program that seeks only to “benchmark” achievement will attract little political support. The second suggests that

the wider agenda of standards-driven educational reform, of which the National

Education Goals are a part, is itself in trouble. Neither contextual point suggests

that there can be much immediate progress in building a national indicator of collegiate attainment—whether from the federal level down, or from the state level

up.

B. Perceived Obstacles and Needs

In addition to asking state respondents to describe their assessment activities, the background papers requested that states describe what they perceived as important obstacles to

assessment, some of their most commonly-encountered methodological problems, and what

they saw as most needed in order to firther a common assessment agenda. A total of twentytwo states provided information of this kind as part of their background papers.

1. Most Important Policy-Level Obstacles. States provided a total of 44 comments

about the most important obstacles to assessment perceived or encountered. In descending order of frequency, these comments clustered around the following

broad categories:

10
.-r.

high costs of development for assessment instruments(11 comments).

institutional resistance, especially from faculty and at research universities (11

comments).

excessive diversity of assessment settings in postsecondary education—both

with respect to mission differences among institutions and variations in the

types of students served; the main theme here was that a single assessment approach would probably not be appropriate across all settings (8 comments).

doubts about the continuing policy utility of assessment+ specially in being

able to communicate complex results to lay audiences in ways that would not

be misinterpreted (6 comments).

lack of the requisite authority on the part of state agencies to undertake assessment in the first place (3 comments).

These obstacles correspond closely to those identified by past inventories of state

assessment activities, and to considerable anecdotal evidence as well.

2. Most Important Methodological Problems. States provided a total of 39 comments to a question about the most important technical or methodological problems that they have encountered or expect in be realm of collegiate assessment.

Here the pattern of cornmenta~ clustered strongly around several themes:

. lack of appropriate instruments; especially cited here were reliable “authentic”

or performance-based instruments, and instruments to assess skills developed

through collegiate general education programs such as critical thinking or effective communication; frequently cited as well were “soft skills” such as interpersonal skills or motivation (16 comments).

. lack of faculty or community consensus about the actual domain to be assessed;

again, skills and attitudes typically associated with undergraduate general education were prominently mentioned under this heading (9 comments).

. a range of program implementation problems including lack of student motivation to perform on non-required tests (3 comments), questions about data reliability and data use (5 comments), and concerns about lack of databases and

state-level expertise in how to analyze assessment data (3 comments).

The overwhelming emergence of instrumentation and domain consensus issues is

not surprising, though considerable progress in the latter had already been made

through the prior series of NCES Study Design Workshops, and ongoing work in

achieving domain consensus on several key skills undertaken by the National Center for Postsecondary Teaching, Learning and Assessment (NCTLA) at Penn

State. Clearly states are not aware of these results or do not believe they have yet

yielded a workable common approach.

3. Most Important Needs. A total of 29 comments were generated about the most

important actions needed to further a common assessment agenda. Again, state

commentary tended to cluster quickly on this matter:

11
● additional funding; this comment, of course, was most often offered by those

citing high costs as an obstacle (10 comments).

. training and staff development; areas especially mentioned here included training for state agency staff in how to develop and use assessment data, training

for faculty and other institutional personnel in how to conduct sound local assessments, and networking among states themselves to share approaches and

ideas (8 comments).

. policy leadership from the federal government+specially in providing common policy direction and definitions (7 responses), and in convincing state legislatures and executive offices that an ongoing assessment agenda is worth

pursuing (4 comments).

Taken together, this pattern of state commentary is consistent with results of the

ACT survey that suggest a high potential state-level demand for assessment

(Steele and Lutz 1995). Most state papers cite primarily operational and logistical

obstacles to proceeding further on this agenda. But results of the 1995 inventory

also make it clear that current state “interest” and “ability to act” in the realm of assessment are still far apart.

111. Discussion Themes and Conclusions - -

Plenary and breakout sessions at the workshop covered a broad array of topics and yielded a number of salient points of agreement. A first set of discussion topics centered on the political context

for assessment and emphasized the ways in which important premises at both the state and federal levels have evolved since the NCES development process began some five years ago. A second set of topics concerned assessment methodology and, more particular y, the degree to which

adequate approaches to postsecondary assessment have, can, and should be developed for use on

a national basis. Finally, the two breakout sessions of the workshop established a number of

points of consensus, as well as a range of conclusions about productive actions that might be

taken.

A. The Political Context

An introduction to the workshop provided by Jean Griffith of NCES-as well as later plenary presentations by Charles Lenth of the Education Commission of the States, Mark Musick of the Southern Regional Education Board, and Ken Nelson of the National Education

Goals Panel-outlined the national policy context within which any contemplated national

postsecondary assessment would have to be crafied. The corresponding state perspective

was provided by Peter Ewell of the National Center for Higher Education Management Systems, and Joe Steele of the American College Testing Program, and “case studies” of specific state efforts in Virginia, Missouri and Tennessee (Banta et. al. 1996). Open discussion

following many of these presentations, as well as deliberations within the various breakout

sessions, helped to clarify this political climate.

All participants were conscious that the initially-contemplated federal role in instrument- development and data-collection would likely be considerably constrained. Nevertheless, national observers advanced a number of continuing reasons why an effort to assess

12
postsecondary outcomes on a national basis would still be needed. Dr. Griffith reasserted the

premise of the workshop by noting that although the federal role appeared to be decreasing,

“questions were still being asked” at all levels about postsecondary performance. As a result, “finding ways for the Department of Education to help states assess student learning”

remained a laudable goal. Mr. Nelson emphasized that the original conditions that led to the

National Goals process still held and that the “standards movement [in K-12 education] is

here to stay.” But he also pointed out a growing political perception that “postsecondary education is lagging in this effort” by not becoming a visible part of increasingly-important

standards-based reform movements at all levels. Sounding a familiar theme, he noted that in

the absence of actions by higher education itself to address this perceived problem, policymakers would not hesitate to apply “blunt instruments.” This position was partially echoed

by Dr. Musick, who observed that a slow but steady increase in interest about academic attainment was developing at all levels. Citing the fact that support for assessment is bipartisan and embraces both supporters and critics of higher education, he predicted that interest

would “cycle back” to these topics in the long run. Recent statements of interest in establishing “rising junior” examinations at the postsecondary level by governors in Arkansas,

Kentucky and Colorado were cited as evidence of this continuing interest.

State representatives acknowledged these concerns, but came from a different policy perspective. As already noted in their background pa~ers,-most states reported that they were

preoccupied with governance and fiscal difficulties, with higher education matters receiving

considerably less attention than they had in the past. One result was that the “quality” issues

driving the establishment of assessment initiatives in the mid-to late-eighties were now being replaced with “productivity” issues that require much more straightforward data to track.

More subtly, state assessment policies had generally not been substantially based on a

“standards-driven” approach in the first place. Instead, like the regional accrediting agencies, these policies emphasized building local assessment capacity at the institutional level

as an aid to instructional improvement. Indeed, a number of state representatives believed

that state policies for higher education in all areas were increasingly moving away from

“standards” and “mandates” to approaches based on “creating markets” and “managing incentives. ” A key factor in ensuring the longevity of Tennessee’s performance funding approach, for instance, was its use of positive incentives and the willingness of state officials

to periodically re-examine standards and measures and revise them as needed (Banta et. al.

1996). Nevertheless, state representatives increasingly recognized the fact that decentralized

assessment approaches, in Dr. Miller’s words, “don’t tell much of a story” to outside audiences. As a consequence, they felt growing pressures to come up with a summative measure

of collegiate learning. As emphasized in their background papers, however, most felt that

the task of creating and fielding such a measure was currently beyond the means of any single state.

All parties agreed that for a number of operational reasons, increased general use of assessment techniques in postsecondary settings was likely. Dr. Musick reminded the workshop

that questions about the effectiveness of remediation, of “bridge” programs among institutions or between high schools and colleges, and of increasingly-prominent technologydriven or distance-learning approaches were of rising policy concern, and that all would

require information about the development of cognitive abilities. Dr. Lenth further emphasized this point by asking participants to imagine a new kind of institution “located next to

13
.-c.

the Information Superhighway” in which competency-based evaluation against clear performance benchmarks would be the norm. Using this vision, he emphasized that changes in

technology and in the socio-political context for postsecondary delivery had moved the policy conversation about assessment far beyond the agenda of “improving undergraduate education” that was typical of the 1980’s.

Despite many differences in perspective, these discussions revealed a number of points of

consensus. Echoing findings of the ACT survey (Steele and Lutz 1995), most agreed that

suitable surnmative measures of student achievement would be useful to states in addressing

a growing accountability problem for postsecondary education. But most also believed that

appropriate measures were not currently available and that developing them was beyond the

means of states acting individually. At the same time, unsettled fiscal conditions and uncertain political will remained major concerns at the state level. As a result, states appeared basically willing to participate in developing some common approaches to measuring

collegiate attainment, but only if such measures were consistent with local policy objectives

of improving productivity and if the expected substantial costs of such an effort could be defrayed by others.

B. The Methodological Context

Methodological perspectives were provided by presentations given by Elizabeth Jones

(Jones and others 1995) of the National Center on Postsecondary Teaching Learning and Assessment (NCTLA), Steven Dunbar of the University of Iowa, John Mazzeo of the National

Assessment of Educational Progress (NAEP) program of the Educational Testing Service

(ETS), and David Lutz of the American College Testing Program (ACT). Additional background on the NAEP and National Adult Literacy Study (NALS) was provided by Gary

Phillips and Andy Kolstad of the NCES staff. Once again, discussions following each of

these plenary sessions provoked a range of additional comments from participants.

From a purely technical standpoint, test designers saw no insurmountable obstacles to building the kinds of assessments required. Experience with both the NAEP and with ACT projects such as “Work Keys” sustain the conclusion that large-scale performance assessments

covering key collegiate skills can be built and that, indeed, prototype measures addressing at

least a part of the intended domain already exist in some form. Problems of more fully specifying this domain—and in such a way that the resulting measures remain credible across a

diverse postsecondary community and with such key external audiences as employers and

policymakers—proved more open to debate. Results of the Delphi-based domain-specification project reported by Dr. Jones-as well as the outcomes of the two previous NCES

Study Design Workshops-do suggest that such consensus is possible. But participants experienced in the actual development of performance assessments at the state level emphasized that even if “domain consensus” of this kind is achieved, it may be far too general.

Instead, the necessary understanding and agreement must occur at the level of actual assessment items and tasks, and results must be communicated in a manner that concretely references actual performance standards and what students are specifically required to do to

attain them. This point was reinforced by experience with the NAEP and NALS, where the

initial establishment of performance levels and public reporting were consistently anchored

in actual test materials.

14
-.

Deeper concerns about the actual uses to be made of such results, however, were widely

voiced by workshop participants. Dr. Dunbar noted, for instance, that well-designed instruments were only one part of an effective assessment technology. Equally required is a surrounding “support structure” of policy and intended use that remains “stable enough” to

give its results validity. Indeed emerging concepts of “validity” demand that the uses of assessment results, and the potential consequences of these uses, be taken fully into account.

From the standpoint of instructional improvement, for example, Dr. Dunbar pointed out that

the domains in question are not explicit but are rather “tacit” objectives of most postsecondary instruction; because few instructors would acknowledge that they are explicitly trying

to develop such skills in college classrooms, attempts to use such results for improvement

will be difficult. Other participants expressed substantial doubts about the real utility of such

results at the state level beyond simple accountability, believing that local measures that

could be linked more closely to actual curricular experiences at different institutions would

be more helpful. Many similar comments emerged as conclusions to the workshop breakout

sessions.

Additional comments offered by workshop participants emphasized other technical features

of the required assessment. One prominent theme was the need for contextual information

about student population characteristics and the attributes of particular instructional settings.

Many felt that contextual information of this kind-was critical to drawing meaningful conclusions about institutional effectiveness, while for improvement purposes it was critical to linking obtained outcomes with specific policies, settings and experiences. Participants also

believed that the most meaningful use of such measures was to show improvement over

time—again emphasizing the need to obtain accurate and appropriate early benchmarks. In

the same vein, most believed that longitudinal designs that could account for differences in

entering student achievement levels, while also examining different paths of student development, would be beneficial. While sample-based assessment designs such as NAEP were

viewed as the most viable approaches to achieving these objectives, some state-level representatives believed that testing all students for mastery was most appropriate-essentially

holding individual students responsible for demonstrating achievement. Such comments reflected previously-mentioned interest in using assessment devices like NAEP for operational purposes-for instance in collegiate admissions. Finally, most agreed that extensive

stakeholder involvement in assessment development was critical for success. As one participant put it, a successful process of development should be neither “bottom-up” nor “topdown” in character. Instead it should involve active face-to-face dialogue in the design

process among the various parties-at-interest.

C. Results of Working Sessions

The workshop’s two working sessions were intended to solicit feedback on a range of topics. A first set of working sessions examined the prospects for meaningfully aggregating existing state-level assessments, as well as the feasibility (or desirability) of developing a

NAEP-like measure of postsecondary attainment for use in common by the states. The second set of working sessions addressed specific steps that might be taken by both the states

and NCES to further a common assessment agenda.

15
-.

1. Ability to Aggregate State Measures. Reports on the feasibility and utility of

state-level common measures were submitted by each of five working groups.

When asked whether existing state-level assessment activities can be used to gain

insight into student learning from a national perspective, the clear consensus of

opinion was that they are currently unable to do so. The reasons most commonly

cited to support this conclusion were that few states use common assessment measures for all institutions, and that even those that do so employ different instruments from state to state. Furthermore, the likelihood of obtaining sufficient

commonality in the future was deemed low. Reasons cited were natural diversities

across states in both policy approach and in the characteristics of their higher education systems. Participants in many groups also pointed out that current statelevel assessment programs generally cover only those students who attend public

colleges and universities, and that any national approach would need to include all

students. Working groups disagreed, however, on the issue of how many states

would be required to draw meaningful conclusions. Some concluded that all states

would have to participate under common guidelines for results to be meaningful at

the national level. Others felt that a useful national picture of collegiate attainment

might be provided if up to a third of states could report outcomes on common dimensions. Finally, one group noted that existing avenues for assessment like regional accreditation and state-level program refiew might be further capitalized

upon; such activities occur regularly for most institutions and the opportunity

might be taken to embed a small number of common assessment measures within

them.

2. Feasibility and Utility of a “NAEP-Like” Measure. When asked whether a national survey of student learning similar to the NAEP or NALS could be used to

meet state assessment needs, conclusions of the working groups were somewhat

more optimistic-though conclusions still remained mixed. Two of five groups

provided a qualified “yes” to this proposition, provided that agreement could be

gained nationally on standards and definitions and that sufficient resources could

be provided. The principal reason advanced was comparative: in the words of one,

“it is important for states to have an idea of how they measure up nationally. ” One

other group concluded that such an approach might be generally useful, but that

states were sufficiently different from one another that they would need to develop their own assessment approaches anyway. The remaining two groups concluded that the results of such an assessment might be “interesting” and “useful as

a benchmark” but that the implementation of such an assessment would not serve

state interests because of widely differing state political conditions, the narrowness of the information that could be derived given the wide variety of outcomes

that might be of interest to different states and institutions, and the likely resistance of institutions to the administration of such an assessment.

Despite mixed responses, the working groups did have some specific suggestions

about how a NAEP-like assessment at the college level might be designed and implemented. Rather than being administered at the end of four years, the suggestion

was made that a sophomore-level test would be the most meaningful because it

would occur at the end of most general education programs in college and could

16
-.

be applied to students attending both two-year and four-year colleges. It was also

widely believed that such an assessment, if developed, should be performancebased and should concentrate on “real-world skills.” Because student motivation

to complete such an assessment was seen as a major obstacle, one group felt that

its content should be seen as “relevant” and “interesting” to students, while another group emphasized that individual feedback should be given to those taking

the assessment. Finally, a suggestion was made that the development of such an

assessment might be evolutionary—initially selecting as a pilot one subject area

that already has a well- defined domain and associated indicators.

3. What Is Needed at the State Level. The second set of working groups addressed

questions about what might be done at the state level to further the assessment

agenda, and what NCES might do to facilitate state-level efforts. Results of these

working groups again highlighted some considerably different points of departure

between state and federal actors. When asked about appropriate state-level action,

most groups again began with the original premise grounding existing state assessment policies: to further institutional planning capacity and to meet growing accountability demands from legislators and the public. As a result, the principal

objects of discussion centered on how to build assessment into already-existing

state policy and quality assurance processes, ~d how to develop effective ways to

induce institutions to participate more filly in assessment activities. Among the

mechanisms most prominently mentioned under the first heading were statewide

academic program review processes (for instance in Illinois or West Virginia). In

a similar vein, many participants urged closer cooperation with regional accreditation-especially working with accreditors to develop more credible and common

standards of performance. Turning to the second, participants emphasized the positive role that state authorities could play in providing a forum for institutions to develop and share assessment approaches-a strategy illustrated by South Carolina’s

state-sponsored but “institutionally-owned” statewide assessment network and

similar initiatives in Virginia and Colorado.

Not surprisingly given this point of departure, the primary suggestions for collective action generated under this heading concerned improved networking and sharing “best practices.” Virtually all the work groups identified as a high priority the

establishment of a formal mechanism for the states to communicate about matters

of common concern in assessment. In most cases, state-level participants agreed,

current cross-state networking efforts sponsored by such agencies as SHEEO were

not adequate because they did not involve the individuals at each agency who are

directly responsible for coordinating assessment. Here information needs of two

kinds were identified. First, practitioners wanted information on documentable

“good practices.” These concerned both the most effective data-gathering approaches available and the most usefid policies to induce institutions to adopt and

use them. Participants felt that both kinds of information could be archived in database form for wider retrieval and analysis. Based on these documented cases,

some participants believed, useful guidelines for the development of assessment

practice under different conditions might be developed for use by all states.

Equally required, others maintained, was a means for state-level assessment coor17
dinators to get directly in touch with one another to seek advice in as close to “real

time” as possible. Quite a number, for instance, cited the NCES workshop itself as

the first opportunity for peer exchange on the topic that they had experienced in

some time. Possible mechanisms suggested to accomplish this objective were electronic communications networks and periodic regional conferences.

4. Actions that NCES Might Take. When asked the question of what NCES might

do to help further state-level assessment activities, participants tended to think in

many of the same terms. Consistent with the information provided in their background papers, they identified funding as the most common single category of assistance. Interestingly, however, the use of finds to develop common instruments

was not among their top priorities. Instead, initiatives aimed at identifying best

practices and at facilitating communication among states again emerged as dominant. Although NCES was not felt to be the only actor that might play this role,

opportunities for building such a network through the National Postsecondary

Education Cooperative were recognized, and NCES was urged to examine these

opportunities fully. Unique to NCES was the related role of providing national

standards and definitions related to assessment. Within the framework of IPEDS,

NCES was felt to be in an excellent position to begin to develop appropriate definitions of key outcomes and standards of good ~ractice in conducting state-based

assessments, even though no common instruments for assessing collegiate achievement could (or should) immediately be contemplated. The role of NCES in shaping state practice in graduation/completion-rate reporting~oth through its drafi

graduation-rate reporting formats and its role in the rulemaking process for “Student Right-to-Know” —were favorably noted in this regard. Once such definitions

and standards were in place, some felt, testing organizations and selected states

would likely move forward to implement them. Consistent with standard-setting,

moreover, some participants suggested that NCES might play a role in “certifying” or “auditing” state assessment practices, much as independent review teams

examine industrial “quality assurance” processes through such mechanisms as the

Malcolm Baldrige National Quality Awards or the international “1S0-9000” process. Working with regional accrediting bodies as an integral part of this process

was again felt to be important.

At the very least, some felt, national coordination might evolve a more common

understanding across states of what actually constitutes “postsecondary assessment” and how the results of assessments should appropriately be communicated

and used. As one participant cogently observed, even after ten years of practice

states still have some very different ideas about the definition and appropriate content of “state-level assessment,” many of which legitimately extend far beyond a

need to track progress on National Goal achievement. Among the broader purposes of this kind explicitly mentioned in working-group sessions were direct

state-level accountability for resources invested, institutional program improvement, state-level priority-setting and policy development, and individual “consumer protection. ”

18
Pervading all comments at the breakout sessions, moreover, was the feeling that stronger national leadership is neededfar more so than better technical expertise. Many state representatives appeared concerned about an erosion of impetus for assessment in their states.

Consistent with their background papers. they believed that the federal government might

help to supply such leadership-a process which had already begun through the National

Goals effort. Participants made it clear that aligning state efforts to create a national picture

of achievement would require an active “impetus for change” at both levels, as well as a set

of well-articulated, compelling reasons for embarking on such an effort. Before discussing

what or how to assess+r deciding on technical matters like instruments and sample sizes—

they felt that all parties must be clear on exactly why the assessment is taking place. In

short, although participants believed that the resulting data might be interesting and would

not mind having such data if direct costs were minimal, the majority of states remained unconvinced that simply “filling a gap in national data collection” provided the necessary rationale for developing an integrated national approach.

IV. Conclusions and Next Steps

A. Some Policy Implications

Anyone involved with the evolution of state-level assessment in higher education over the

last decade will discover many fhrniliar themes in these Proceedings. Indeed, several workshop participants confessed to a feeling of “deja vu” when discussing this topic, and noted

that they had engaged in debates about most of its aspects many times before. Among the

specific issues mentioned in this regard were the necessity of fully stating and achieving consensus on the purposes of assessment before embarking on the technology of instrument design, the complexity of achieving agreement on the domain to be measured (as well as the

wisdom of concentrating attention at the “item-level” in such discussions), and the difficulty

of designing appropriate instruments that reflect authentic performances reliably but whose

results can be summarized succinctly and understandably.

Despite the perception of having “heard it all before,” workshop participants acknowledged

that the circumstances under which these matters were being discussed had changed significantly. First, it was clear that immediate funding from either states or from federal sources

to support a national assessment effort is unlikely to be available. As a result, any alternatives discussed would have to use existing vehicles and/or rely upon joint ventures and cooperative efforts through which current resources might be leveraged. At the same time,

substantial political uncertainties at both the state and the federal levels mean that it is

equally unlikely that a clear policy direction will quickly emerge. This makes it all the more

imperative that if something is to be done, that it be done through established organizational

structures.

Second, most states face pressing short-term demands to demonstrate the effectiveness of

their higher education investments. As a result, many would welcome a straightforward way

to document collegiate attainment using a common yardstick, were such an instrument available. This stance, of course, represents a significant shift from the dominant policy position

of a decade ago, when only a few states were willing to contemplate the use of common

measures.

19
>

Third, assessment technology has progressed steadily during this period—substantiating the

premise that credible, valid, and reliable performance-based approaches can in fact be developed at the postsecondary level. Experience with the NAEP and NALS, state- levei track-records in developing comprehensive assessments like the New Jersey “Assessment of

General Intellectual Skills (GIS),” and the growing sophistication of test-makers in the development of performance-based instruments at the K- 12 level, all suggest that if sufficient

funding and political capital were provided, and clear purposes for the effort established, appropriate instruments could in fact be built.

Fourth and finally, significant changes are impending in the way business is done in postsecondary education that promise to render the use of assessment technologies more prominent

and legitimate. Among the developments noted here were the use of competency-based admissions and articulation standards and the need to certify or evaluate learning obtained

through alternative instructional-delivery mechanisms. As such developments progress, they

will likely stimulate the use of more common standards and measures.

Taken collectively, these substantial changes in context lead to a somewhat different set of

policy answers to the “same old questions.” From the technical standpoint, participants in

the workshop agreed, a standard national approach to collegiate assessment can be crafted.

The issues remaining are: why would states want such an instrument and for what? Essentially, results of the workshop suggested three possi%le policy answers, with quite different

implications about the shape of the required assessment.

. standard assessments of collegiate attainment are needed to meet increasingly pressing state-level accountability demands. In the short term, the most immediate use for a

summative assessment of collegiate skills is to demonstrate effectiveness on a state-bystate basis. Because such demands are generated within each state-and not all states are

facing such demands—use of the same measure by every state would not be required.

Such commonality might be desirable to benchmark state results, however. Given this

purpose, a sample-based approach would be adequate because only aggregate reporting

would be needed.

. standard assessments of collegiate attainment are necessary to both drive and inform state-level improvement efforts. Informing improvement and guiding instructional

re-structuring and reform remain the primary reasons why states initially engaged in assessment—though they have done so on a decentralized basis. Nothing in this objective

requires the use of similar measures across states and sample-based, aggregate-level measures would be sufficient to meet it. Equally required in this case, however, would be two

important additional conditions: data about contexts and instructional “good practices”

that could be correlated with outcomes, and real linkages to policies and resources (both

state and federal) that would induce states and institutions to act on the basis of what is

found.

s standard assessments of collegiate attainment will be increasingly needed in order to

actually run the postsecondary enterprise. This policy answer reflects the growing possibility that assessment technology will be directly embedded in state postsecondary operations in such areas as assuring college readiness, in certifying specific levels of

attainment, or in regulating transferability among institutions. Unlike the other two policy

20
options, the technology suited

ble students, not just a sample

to such applications would have to be applied to all applicaBut construction of the required instruments might follow

design principles similar to those used in NAEP. Operational-level results could then be

aggregated for indicative purposes at the state and possibly national levels. A few existing

state assessment programs—for instance the College Level Academic Skills Testing

(CLAST) in Florida and the Texas Academic Skills Program (TASP) in Texas-already

embody aspects of this “operational” application.

Given these alternatives, and the current dearth of funds for instrument development, states

would likely welcome a standard approach as a common benchmark for comparison if they

could gain access to it at little cost and if it did not obstruct local policy objectives. Few,

however, would adopt such an approach as a substitute for their current institution-focused

programs or as a key component of state-level policy information.

B. Some Possible Next Steps

Results of the workshop indicate that there is no immediate possibility of using existing

state-level measures to help construct a national picture of collegiate attainment. They further suggest that there are few prospects of states quickly developing such a capacity. But

the workshop did yield a number of useful suggestions ajout how the states and NCES

might work together more effectively to further a common assessment agenda. Among these

potential next steps are four specific areas of potential cooperation.

1< Consortia of states—perhaps acting in partnership with NCES—might adopt

an available instrument for use in common. Several states already use a standardized instrument to assess collegiate general education. And despite doubts

about the validity and utility of these measures, a growing number of states express interest in fielding such an instrument purely to satisfy imperative demands

for short-term accountability. As a result, support for using an available instrument in common might be fostered among a self-selected group of states—both to

demonstrate effectiveness and to help benchmark their own institution-centered assessment efforts. Especially attractive in this regard might be regional consortia

such as the Southern Regional Education Board (SREB) or the Western Interstate

Commission on Higher Education (WICHE) where contextual differences among

states might be minimized. Another alternative would be for states to collectively

investigate a common problem or domain, using locally-chosen or individually-developed instruments as appropriate. Examples here might include minority or rural student achievement or, consistent with Goal 3, collegiate attairunent in math

and science. Under the former alternative, NCES might foster such collaboration

through the National Postsecondary Education Cooperative (NPEC). Under the latter, NCES and the states might actively identi~ and formulate a research topic for

collective investigation, help develop an appropriate research design, and help coordinate and calibrate data-collection efforts across states.

2. NCES might extend coverage of the National Assessment of Educational Progress (NAEP) to a national sample of college students. Though this alternative

would contribute little toward current state agendas, it has the substantial virtue of

21
3.

using an existing vehicle for data-collection. While legitimate objections can be

raised that an instrument such as NAEP is not capable of reflecting the full range

and sophistication of collegiate domains, results of the NALS suggest that current

college graduates have not “topped out” on far more basic skills, and that a national benchmark would be useful. At the same time, administration of the NAEP

to a sample of college students might help determine the feasibility of wider datagathering at the state Ievel+specially with regard to such intangibles as student

motivation, faculty and institutional cooperation, and how the results of such an effort might be interpreted and used. As above, the initiative could be focused initially on a single selected domain area. Results of the previous two NCES Study

Design Workshops imply that assessing the domain of critical thinking/problemsolving is feasible. But more immediate progress might be made in areas like collegiate-level communication or quantitative skills where a good deal of domain

consensus and a body of widely-credible assessment technology already exist.

Choosing the latter area would have the additional benefit of informing Goal 3.

NCES, in partnership with other appropriate national bodies or associations,

might embark upon a formal program of networking and information-sharing about assessment among state-level postsecondary assessment practitioners. The need for better lines of communication about current events and “best

practices” in state-level assessment was apparent throughout the workshop.

Mechanisms suggested included not only further conferences, but also use of the

Internet and alternative forms of information exchange. Whatever the medium, all

agreed that the focus should be placed upon substantive questions of policy rather

than simply “assessment technology. ” Who should (or could) sponsor such a network, however, remains an open question. Organizations like AAHE and SHEEO

are in many ways more logical “sponsors” for such networking than is NCES—

and indeed are already partially engaged in such activities. But workshop participants felt currently underserved by such organizations. Partnering with NCES

through the establishment of a standing subcommittee on student outcomes as an

integral part of the emerging NPEC might keep such organizations focused on

serving those responsible for state assessment policy. Directly convening these individuals on a regular basis through an arrangement similar to NCES’ annual Network Data Conference (or creating special session tracks within that conference)

might also constitute viable options.

4. NCES might attempt to proactively shape state assessment activities by evolving a framework of standards, expectations, and “best practices” for stateIevel assessment. The primary precedent for this activity is NCES’ role in the

development of standard practices for calculating and reporting postsecondary

completion rates; despite the fact that no “standards” were eventually mandated in

this arena, actual state practices have converged around definitions and procedures

originally developed at the federal level. Analogous opportunities in the realm of

state-level practice in collegiate assessment include the technical design of summative assessments of collegiate achievement, specific guidelines on the domains

and sample characteristics that would render individual state-level efforts both

valid for local purposes and able to contribute to a national portrait of collegiate

22
attainment, and advice about effective practice in the public reporting of assessment results. Again. the emerging NPEC, in partnership with ECS and SHEEO,

would provide a natural organizational framework for such an activity.

While such steps are modest in comparison to construction of a national indicator of postsecondary attainment, they would arguably be as useful in fostering the development of effective practice in this important area. And results of the workshop strongly suggest that

assessment remains important to the states, despite changed conditions. Sustained grassroots

efforts to improve its practice may therefore pay substantial long-term dividends, both to individual states and to the nation as a whole.

.

23
>

REFERENCES

13anta, T. W., Rudolph, L. B., Van Dyke, J. V., and Fisher, H.S. (1996). Performance Funding

Comes of Age in Tennessee. Journal oj’ Higher Education, V67, #1, January/February 1996,

23-45.

Boyer, C. M., Ewell, P. T., Finney, J. E., and Mingle, J.R. (1987). Assessment and Outcomes Measurement: A View from the States. AAHE Bulletin, 39(7), 8-12.

Ewell, P. T., Finney, J. E., and Lenth, C. (1990). Filling in the Mosaic: The Emerging Pattern of

State-Based Assessment. AAHE Bulletin, 42(8), 3-7.

E\vell, P.T. and Boyer, C.M. (1988). Acting Out State-Mandated Assessment: Evidence from

Five States. Change, 20(4).

Jones, E. A., and others (1995). National Assessment of College Students Learning: Identlfiing

College Graduates’ Essential Skills in Writing, Speech, Listening, and Critical Thinking.

Washington, DC: U.S. Department of Education, Office of Educational Research and Improvement (OERI).

Ruppert, S.S. (1996). The Politics of Remedy: State Legislative Views on Higher Education.

Washington, DC: National Education Association.

Ruppert, S. S., ed. (1994). Charting Higher Education Accountability: A Sourcebook on StateLevel Performance Indicators. Denver, Co: Education Commission of the States (ECS).

Russell, A. B., and Chisholm, M.P. (1995). Tracking in Multi-Institutional Contexts. P. Ewell

(cd) Student Tracking: New Demands, New Techniques, New Directionsfor Institutional

Research #87, Fall 1995,43-54.

Steele, J.M. and Lutz, D.A. (1995). Report of ACT’s Research on Postsecondary Assessment

Needs. Iowa City: American College Testing Program, June 1995.

24
APPENDIX A-1

WORKSHOP AGENDA

%

25
1995 NATIONAL CENTER FOR EDUCATION STATISTICS

POSTSECONDARY EDUCATION ASSESSMENT PLANNING

WORKSHOP

December 7-8, 1995

Key Bridge Marriott

Arlington, VA

R 7,1995

8:00-9:00: Breakfast Meeting of Workshop Team

8:30-9.00: Continental Breakfast for all Participants

9:00-9:05 Opening Session (Introduction) - Sal Corrallo, NCES Project Director,

National College Student Learning

9:05-9:30: Keynote Speaker - Jeanne E. Griffith, Acting Commissioner, NCES - Purpose

and Expectations - (Group) .

9:30-11:30: Morning Session - “Current State Assessment Activities”

Moderator: Robert Wallhaus, State Higher Education Executive Officers

Presenters: Peter Ewell, NCHEMS - Summary of papers prepared by state participants

Joe Steele, ACT - Research on Assessment Needs

Trudy Banta, IUPUI - Performance Assessment in the States

Questions and Comments

11:45-1:15: Luncheon - “National & State Assessment in the Current Political Climate”

Moderator: Charles Lenth, Educational Commission of the States

Speakers: Ken Nelson, The National Perspective

Mark Musick, The State Perspective

1:30-3:15: First Afternoon Seminar -” What Can Be Assessed at the State Level?”-

Moderator: Pat Yaeger, NCPTLA

Presenters: Beth Jones, NCPTLA - National Education Goals Skills To be Taught, Learned,

and Assessed .

Peg Miller, - The Virginia Experience

Ava Fajen, Richard Stein, and Charles Kupchella - Making Assessment

Without Legislative Mandates: The Missouri Experience

Questions and Comments

3:15-3:30: Break

26
1995 NATIONAL CENTER FOR EDUCATION STATISTICS

POSTSECONDARY EDUCATION ASSESSMENT PLANNING

WORKSHOP

December 7-8, 1995

Key Bridge Marriott

Arlington, VA

3:30-5:30: Second Afternoon Seminar - “Methocfo/ogica/ Considerations”

Moderator: James Ratcliff, NCPTLA

Presenters: Steve Dunbar, University of Iowa -An Overview of Issues and Concerns

David Lutz, ACT - The ACT Assessment Approach

John Mazzeo, ETS - The ETS Assessment Approach

Questions and Comments

7:30-9:30: Working Dinner in Hotel - Table groups to consider two questions.

Leader/Recorder to be named. - -

DAY, J3FCE.MBER & 1995

8:00-8:30:

8:30-9:15:

9:15-10:30:

10:30-10:45:

10:45-11:30:

11:30-12:15:

12:15:

12:30:

12:45-2:30:

Breakfast

Brief notes from dinner table discussions.

Small Group Sessions - Participants Assigned at Random to each group.

Check board for room. Each session will consider three questions.

Group 1- Robert Wallhaus

Group 2- Charles Lenth

Group 3- James Ratcliff

Group 4- Patricia Yaeger

Break

Breakout Reports

Open Mike

Closing Comments - Peter Ewell, NCES

Adjournment

Working Lunch for Workshop Team and Invited Participants

27
.
?

APPENDIX A-2: PARTICIPANT LIST

Thomas Angelo. Director

AAHE Assessment Forum

One Dupont Circle

Suite 360

Washington, DC 20036

(202)293-6440 Ext. 29

(202)293-73 Fax

Trudy Banta

Vice Chancellor

IUPUI

Administration Building 140

355 North Lansing Street

Indianapolis, IN 46202-2896

(3 17)274-4111

(317)274-465 1 Fax

Robert Barak

Deputy Executive Director

State Board of Regents

Old Historical Building

East 12th and Grand Avenue

Des Moines, IA 50319

(515)281-3934

(5 15)281 -6420 Fax

Diane Barrans

Executive Director

Alaska Commission on Postsecondary

Education

3030 Vintage Boulevard

Juneau, AK 99801-7109

(907)465-6740

(907)465-3293 Fax

Sandy Bennett

Director, Academic Evaluation and

Assessment

South Dakota State University -

South Dakota Board of Regents

Pugsley201

BOX 2218

Brookings, SD 57007

(605)688-4217

(605)688-63 13 Fax

David Boesel

Acting Director

Department of EducatiotiPLLI

555 New Jersey Avenue, NW

Washington, DC 20208

(202)219-2207

(202)21 9-2030 Fax

Joann Carter-Wells

Professor of Reading/Coordinator,

Undergraduate Reading Program

California State University, Fullerton

20022 Femglen Drive

Yorba Linda, CA 92686

(714)773-3357

(714)773-33 14 Fax

Melodie Christal

Director

SHEEO&CES Communication Network

State Higher Education Exe. Officers

707- 17th Street

Suite 2700

Denver, CO 80202-3427

(303)299-3688

(303)296-8332 Fax

Jose Colon de la Matta

Official Executive

University of Puerto Rico

Office of Academic Planning

23344 UPR Station

San Juan, PR 00931-3344

(809)764-0000 Ext. 3448

(809)764-2822 Fax

Peter Consacro

Associate Vice Chancellor

for Academic Affairs

Tennessee Board of Regents

1415 Murfreesboro Road

Suite 350

Nashville, TN 37217

(61 5)366-4400

(615)366-4464 Fax

29
Sal Corrallo

Project Director, NACSL

NCES

555 New Jersey Avenue. NW

Room 308E

Washington, DC 20208

(~()~)~19.]913

(202) 219-1 879 Fax

Gary Cox

Executive Director

Council on Higher Education

1024 Capital Center Drive

Suite 320

Frankfort, KY 40601

(502)573-1555

(502)537-1537 Fax

Faith Crampton

Program Principal

National Conference for State Legislators

1560 Broadway

Suite 700

Denver, CO 80202

(303)830-2200

(303)863-8003 Fax

Stephanie Cunningham

Academic Officer

Colorado Commission on Higher

Education

1300 Broadway

2nd Floor

Denver, CO 80202

(303)866-2723

(303)860-9750 Fax

Maria del R. Rodriguez Orellana

Director

University of Puerto Rico

Office of Academic Planning

23344 UPR Station

San Juan, PR 00931-3344

(809)764-0000 Ext. 3448

(809)764-2822 Fax

-.

Eusebio Diaz-Diaz

Director. Policy and Planning Division

Council IHigher Education. PR

Box 23305

UPR Station

San Juan. PR 00931

(809)758-3350

(809)725-2150 Fax

Barbara Doersom

Acting Commissioner for Postsecondary

and Higher Education

Pennsylvania Department of Education

333 Market Street

Harrisburg, PA 17126-0333

(717)787-5041

(717)783-0583 Fax

Steve Dunbar

Professor -

The University of Iowa

224B Lindquist Center

Iowa City, IA 52242

(319)335-5561

(3 19)335-6038 Fax

Odus Elliott

Academic Officer

Coordinating Commission for

Postsecondary Education

Title 11

P.O. Box 95005

Lincoln, NE 68509-5005

(402)471 -0023

(402)471-2886 Fax

Penelope Engel

Professional Associate

Educational Testing Service

1776 Massachusetts Avenue, NW

Suite 620

Washington, DC 20036

(202)659-8043

(202)659-8075 Fax
Karl Engelbach

-.

David Gardner

Senior Fiscal and Policy Analyst

California Postsecondary Education

Commission

1303 J Street

Suite 500

Sacramento, CA 95814

(916)322-7331

(916)327-4417 Fax

T. Dary Erwin

Director of Student AssessmentiProf.

of Psychology

James Madison University

Cardinal House

Harrisonburg, VA 22807

(540)568-6706

(540)568-7878 Fax

Peter Ewell

Senior Associate

NCHEMS

P.O. Box P

Boulder, CO 80301-9752

(303)497-0371

(303)497-0338 Fax

Ava Fajen

Research Associate

Missouri Coordinating Board for

Higher Education

3515 Amazonas Street

Jefferson City, MO 65109-5717

(3 14)75 1-2361

(314)75 1-6635 Fax

Carol Fuller

Assistant VP for Research& Policy

Analysis

Nat’l. Assoc. of Independent Colleges

& Universities

1025 Connecticut Avenue, NW

Suite 700

Washington, DC 20036-5405

(202)785-8866

(202)835-0003 Fax

Deputy Assistant Commissioner for

Research, Planning and Finance

TX Higher Education Coordinating Board

P.O. BOX 12788

Capitol Station

Austin, TX 78711

(512)483-6150

(512)483-6169 Fax

Donald Goss

Director of Academic Affairs

Tennessee Higher Education

Commission

404 James Robertson Parkway

Suite 1900

Nashville, TN 37243-0830

(6 15)741 -7565

(61 5)741-6230 Fax

Jeanne Griffith

Acting Commissioner

NCES

555 New Jersey Avenue, NW

Room 400F

Washington, DC 20208

(202)219-1828

(202)2 19-1736 Fax

Joe Hagy

Director of Special Programs

Oklahoma State Regents for Higher

Education

500 Education Building

State Capitol Complex

Oklahoma City, OK 73105-4500

(405)524-9154

(405)524-9230 Fax

Martine Harnmond-Paludan

Executive Director

New Jersey Commission on Higher

Education

CN542

Trenton, NJ 08625

(609)292-43 10

(609)292-7225 Fax
Henry Hector

Executive Director

AL Commission on Higher Education

P.O. Box 302000

Montgomery, AL 36130-2000

(334)242-2123

(334)242-0268 Fax

Cel Johnson

Director, Institutional Research

Montana State University

212 Montana Hall

Bozeman, MT 59717

(406)994-4390

(406)994-2893 Fax

Elizabeth Jones

Research Associate

Pennsylvania State University

National Center on Postsecondary

Teaching, Learning & Assessment

403 South Allen Street

Suite 104

University Park, PA 16801-5252

(814)865-6346

(814)865-3638 Fax

Richard Jensen

President

Western Interstate Commission for

Higher Education

PO Drawer P

Boulder, CO 80301-9752

(303)541-0205

(303)541-0291 Fax

Charles Karelis

Director

FIPSE

Seventh and D Streets, SW

Room 3100

Washington, DC 20202

(202)708-5750

(202)708-61 18 Fax

Michael Keller

Director of Policy Analysis and Research

Maryland Higher Education Commission

16 Francis Street

Annapolis, MD 21401-1781

(41 0)974-2971

(41 0)974-5994 Fax

Kathleen Kelly

Deputy Director, Academic Affairs

Illinois Board of Higher Education

4 West Old Capitol Plaza

Room 600.

Springfield, IL 62701

(217)782-3442

(217)782-8548 Fax

Judy Kemp

Vice President for Academic Affairs

University of Maine at Machias

University of Maine System

9 O’Brien Avenue

Machias, ME 04654

(207)255-33 13 Ext. 224

(207)255-4864 Fax

Gene Kemper

Vice Chancellor for Academic Affairs

North Dakota University System

State Capitol

600 East Boulevard Avenue

Bismarck, ND 58505-0230

(701)328-2960

(701 )328-296 1 Fax

Andrew Kolstad

National Center for Education Statistics

555 New Jersey Avenue, NW

Washington, DC 20208-5653

(202)2 19-1773

(202)2 19-1801 Fax
Roz Korb

CSSB

555 New Jersey Avenue

Room 311C

Washington, DC 20208

(202)219-1587

Alan Krech

Assoc. Comm. of Planing Assess. & Voc./

Tech. Education

South Carolina Commission on Higher

Education

1333 Main Street

Suite 200

Columbia, SC 29201

(803)737-2291

(803)737-2297 Fax

Charles Kupchella

Provost

Southeast Missouri State University

1 University Plaza

Cape Girandeau, MO 63701

(3 14)65 1-2063

(3 14)651-5102

Charles Lenth

Director of Policy Studies Higher

Education

Education Commission of the States

707 17th Street

Denver, CO 80202

(303)299-3667

(303)296-8332 Fax

Barbara Lieb

Senior Researcher/Postsecondary Education

OEFU

555 New Jersey Avenue

Washington, DC 20208

(202)2 19-2191

(202)219-2030 Fax

David Lutz

Director, Postsecondary Services

American College Testing

2201 North Dodge Street

P.O. Box 168

Iowa City, IA 52243-0168

(319)337-1051

(319)337-1 790 Fax

Charles Manning

Chancellor

University System of West Virginia

1018 Karmwha Boulevard East

Suite 700

Charleston, WV 25301

(304)558-2736

(304)558-3264 Fax

John Mazzeo

Director of Scoring, Operations and

Reporting

Center for the Assessment of Educational

Progress/ETS

Rosedale Road

Princeton, NJ 08541

(609)734-5298

(609)734-1 878 Fax

Thomas McCord

Coordinator, Office of Research&

Information Systems

New York State Education Department

Room 5B44

Cultural Education Center

Albany, NY 12230

(51 8)474-6644

(5 18)474-1907 Fax

Margaret Miller

Associate Director for Academic Affairs

Virginia State Council of Higher

Education

James Monroe Building, 9th Floor

101 North 14th Street

Richmond, VA 23219

(804)225-2627

(804)225-2604 Fax
Sunny Minedew

Senior Research Analyst

University & Community College System

of NV

2601 Enterprise Road

Reno, NV 89512

(702)784-4022

(702)784-1 127 Fax

Joseph Moore

Director of Planning and Academic

Affairs

Vermont State Colleges

P.O. Box 359

Waterbury, VT 05676

(802)241-2520

(802)241-3369 Fax

Edward Morante

Dean, Educational Resources, Research

& Technology

College of the Desert

43-500 Monteret Avenue

Palm Desert, CA 92260

(619)773-2556

(619)586-5955 Fax

Mark Musick

President

Southern Regional Education Board

592 Tenth Street, NW

Atlanta, GA 30318-5790

(404)875-921 1

(404)872-1477 Fax

Helen Myers

Assistant Vice President of Academic

Affairs

University of Alaska

Suite 202 Butrovich Building

P.O. Box 7554500

Fairbanks, AK 99775-5400

(907)474-6302

(907)474-7570 Fax

Ken Nelson

Executive Director

National Education Goals Panel

1255 22nd Street, NYk7

Suite 502

Washington, DC 20037

(202)632-0952

(202)632-0957 Fax

Betty Overton

Coordinator for Higher Education

W.K. Kellogg Foundation

1 East Michigan Avenue

Battle Creek, MI 49017

(616)968-1611

(616)968-0413 Fax

Gerald Patton

Executive Associate Director

Commissicm on Higher Ed./Middle States

Association

3624 Market Street

Philadelphia, PA 19103

(2 15)662-5606

(21 5)662-5501 Fax

Gary Phillips

Associate Commissioner

Education Assessment, NCES

555 New Jersey Avenue, NW

Room 408C

Washington, DC 20209

(202)2 19-1761

Paul Planchon

Associate Commissioner

NCES

Education Surveys Division

555 New Jersey Avenue, NW

Room 413B

Washington, DC 20208-5652

(202)2 19-1614

(202)219-1 728 Fax
-.

Susan Poulsen

Program Officer in Education

The Johnson Foundation. Inc.

P.O. Box 547

Racine, WI 53401-0547

(414)681-3341

(414)681-3325 Fax

William Proctor

Executive Director

Postsecondary Education Planning

Commission

Florida Education Center

Tallahassee, FL 32399

(904)488-7894

(904)922-5388 Fax

Anne Quinn

Education Policy Analyst

Governor Romer’s Office

136 State Capitol

Denver, CO 80203

(303)866-4849

(303)866-2003 Fax

Marilyn Quinn

Executive Director

Delaware Higher Education Commission

820 North French Street

4th Floor

Wilmington, DE 19801

(302)577-3240

(302)577-6765 Fax

Karen Rasmussen

Associate Commissioner for Policy

& Planning Studies

Commission for Higher Education

101 West Ohio Street

Suite 550

Indianapolis, IN 46204-1909

(3 17)464-4400

(317)464-4410 Fax

James Ratcliff

Professor and Director

Center for the Study of Higher Education

& NCPTLA

403 South Allen Street

Suite 104

University Park, PA 16801-5252

(814)865-6347

(814)865-3638 Fax

Santee Ruffin, Jr.

1276 North Wayne Street

#1209

Arlington, VA 22201

(703)8 12-8646

Phyllis “Teddi” Saxman

Assistant Commissioner for Academic

Affairs

Utah Sys@m of Higher Educ.- Board of

Regents

355 West North Temple

3 Triad Center, Suite 550

Salt Lake City, UT 84180-1205

(801)321-7119

(801)321-7199 Fax

Nancy Schamz

SHEEO Project Officer

NCES

555 New Jersey Avenue, NW

Room 315A

Washington, DC 20208-5652

(202)219-1 590

(202)219-1679 Fax

Bill Simpson

Deputy Director for Educational Programs

New Mexico Commission on Higher

Education

1068 Carrillos Road

Santa Fe, NM 87501

(505)827-7383

(505)827-7392 Fax
Catherine Smith

Coordinator

Michigan Department of Education

Michigan Educational Assessment

Program

P.O. BOX 30008

Lansing, MI 48909

(517)335-0564

(517)335-1 186 Fax

Suzanne Snyder

Chancellor

Univeristy of West Virginia System

1018 Kanawha Boulevard, East

Suite 700

Charleston, WV 25301

(304)556-2736

(304)556-3264 Fax

John Spraggins

Deputy Director for Academic Affairs

Arkansas Department of Higher Education

114 East Capitol

Little Rock, AR 72201

(501)324-9300

(501)324-9308 Fax

Joe Steele

Director, College Outcome Measures

ACT

P.O. Box 168

Iowa City, IA 52243

(319)337-1121

(319)339-3020 Fax

Joe Szutz

Assistant Vice Chancellor for Planning

Board of Regents, University System of GA

244 Washington Street, SW -

Atlanta, GA 30334

(404)656-2213

(404)657-41 30 Fax

:..

Kathe Taylor, Ph.D.

Senior Policy Associate

Higher Education Coordinating Board

917 Lakeridge Way

Olympia, WA 98504

(360)753-7815

(360)753-7808 Fax

Kenneth Tolo

Senior Advisor to the Secretary

U.S. Department of Education

600 Independence Avenue, SW

Room 5164

Washington, DC 20202-8100

(202)205-0706

(202)401-3075 Fax

Robert Wallhaus

Senior Consultant

SHEEO .

2712 Lakepoint Tower

505 North Lakeshore Drive

Chicago, IL 60611

(3 12)245-9390

(312)222-1 126 Fax

E. Garrison Walters

Vice Chancellor for Academic &

Access Programs

Ohio Board of Regents

30 East Broad Street

36th Floor

Columbus, OH 43266-0417

(614)466-0885

(614)466-5866 Fax

Cynthia Ward

Associate Commissioner

Rhode Island Office of Higher Education

301 Promenade Street

Providence, RI 02908

(401)277-2685

(401)277-61 11 Fax
Kareen Wenk

Education Specialist

Department of Education/OPE/PPI

7th and D Streets. SW

Room 4050

Washington, DC 20202-5121

(202)708-8875

(202)708-91 07 Fax

Ellen Wert

Program Officer

The Pew Charitable Trusts

2005 Market Street

Suite 1700

Philadelphia, PA 19103

(215)575-4832

(215)575-4939 Fax

Karrin Wilks

Asst. Director of Planning & Academic

Affairs

Vermont State Colleges

P.O. Box 359

Waterbury, VT 05676

(802)241-2520

(802)241-3369 Fax

Patricia Yaeger

Graduate Research Assistant

Penn State University

Center for Study of Higher Education

405 South Allen Street

Suite 104

University Park. PA 16801-5252

(814)863-0852

(814)865-3638 Fax
.
APPENDIX A-3: BACKGROUND PAPER

1995 NCES POSTSECONDARY EDUCATION ASSESSMENT

PLANNING CONFERENCE

Background: The National Center for Education (NCES) was reauthorized in 1994. Section 410,

Cooperative Education Statistics Systems, states that:

The Commissioner may establish one or more national cooperative education statistics

systems for the purpose ofproducing and maintaining, with the cooperation of the states,

comparable and useful information and data on elementary and secondary education,

postsecondary education, and libraries, that are useful for policymakers at the federal,

state, and local levels, In carrying out this section, the Commissioner may provide technical assistance, and make grants and enter into cooperative agreements.

The same act also included the Goals 2000 legislation which calls for improving the effectiveness

and efficiency of learning at all levels of education. Of particular concern to NCES are Goal 3

and Goal 6 (formerly Goal 5) as cited in the legislation. Both call for the enhance- ment of learning of communication and thinking skills among students at the K-12 and college levels. NCES,

as the primary agency concerned with the collection of data on the education industry, is expected to provide a fair and accurate assessment of student achievement at all levels. The National Assessment of Educational Progress (NAEP) is cited in the legislation as the primary

source of information for students in K-12. And although at the postsecondary level, the authorizing legislation provides that NCES is to collect, analyze and disseminate statistics on student

achievement at all levels of education.

Collecting assessment information because it is authorized is but one reason for developing an assessment program. In light of maj or economic and social changes that have and are occurring

both within the nation and the world community, students and parents alike consider a quality

higher education a key component to obtaining and holding a job in the dynamic world today.

Thus consumers are increasingly interested in obtaining information on the quality of the education level that schools and colleges must provide, not withstanding the increasing cost of obtaining a higher education. As a result state legislators and community leaders are increasingly called

to assess higher education, which in turn has put increased pressure on state governing bodies for

the assessment of student learning. These concerns had a large part in encouraging the development of the National Education Goals, formulated first by the nation’s governors and later

adapted by Presidents Bush and Clinton.

Current Activities: In direct response to the adoption of the National Education Goals, NCES

has had an active program explorjng ways to effectively assess the set of skills identified under

Goal 6 Objective 5; higher order communication, problem solving, and critical thinking skills.

Two planning workshops were conducted: the first, in 1991, to identify issues and concerns and

second, in 1992, to identify the steps necessary to begin the process of skills identification. Both

workshops provided a number of working papers. Collectively, participants in both workshops

supported the notion that there are three sets of interrelated activities necessary for the achievement of each objective and for fulfillment of an effective assessment process. These include:
1.

2.

3.

Identification of the Knowledge, Skills, and Abilities (KSA’S) Students Need to Know:

There needs to be agreement as to what is to be taught and learned and then assessed from

the perspective of faculty and staff, the larger community of citizens and the business sector.

In particular, what knowledge, skills, and abilities are needed and at what level of achievement?

Enhancement of Teaching/Learning Functions (TLF): In designing an assessment, consideration needs to be given as to how the information collected through the assessment

process might be used to enhance the teaching/learning of each of the skills. Further, the assessment instrument should accurately capture what students are being taught.

The Implementation of the Assessment Process: Information collected as part of the assessment process may be used for accountability, fiduciary or legal requirements, program

improvements, school and faculty performance, and funding decisions. However, the purpose of assessment must be well understood from the beginning of the design process. It is

also at this point that consideration must be given to alternative approaches given academic,

political, social and economic constraints that affect the assessment process.

In 1993, NCES prepared a Request for Proposal (RFP) that focused upon skills identification for

the college graduate. The purpose was to identify and obtain a consensus of the skills and levels

of competency needed by college graduates beyond college. The results were to be used as a basis for the development of a national survey of college student reaming. A primary use of the information was to be used on the status of efforts to meet the national education goals. The RFP

was not funded due to budget constraints and it is unlikely that NCES will soon have sufficient

resources to conduct a comprehensive national study, as originally envisaged. As a result, alternative strategies were developed.

The Penn State Projects: Over the 1993,1994, and 1995 fiscal years, NCES has funded three

studies to identify the core skills needed by college graduates in the workplace. In FY 1993, the

National Center for Postsecondary Teaching, Learning, and Assessment (NCPTLA) at Penn State

conducted a two-stage Delphi study to identify the writing, speaking and listening, and critical

thinking skills needed by college graduates. Three panels, each with 200 members from the academic and business communities, served as judges for the Delphi study. The final report was published in July 19951. A similar study was funded in FY 1994 for problem solving, which is

currently underway. A reading study was funded in FY 1995 and will begin in late 1995, This

work will provide the initial listing of skills needed by college graduates as cited under the Goals

2000 legislation. However, no fiulher work is planned beyond these three projects.

National Job Analysis Study: Over the same time period, NCES has been working with the Department of Labor to define a generic set of job skills in high performance workplace settings

from a national perspective. This project was and is much more job specific. Using a national

sample of workers, the first step was to identifi sets of specific tasks and the needed skills based

upon specific job experiences. This is in contrast to the judgmental approach used in the

NCPTLA project. Although the original intent was to identify the skills needed by high school

1 Jones, E.A. and others., National Assessment of College Students Learning: Identifying College

Graduates Essential SkIlls in Writing, Speech, Listening, and Critical Thinking. U.S. Department

of Education, Office of Educational Research and Improvement, National Center for Education

Statistics, Washington, D. C., July 1995 (NCES 95001a).

40
graduates, the study will provide a generic set of job skills. Once these have been identified, appropriate assessment instruments will be developed. At that time consideration will be given to

the means of assessing the attainment of the skills from a national perspective. The potential vehicle for a national assessment and the expected costs are unknown at this time. A final report is

not expected to be available until at least the end of 1996.

National Adult Literacy Study (NALS): A national assessment of adult literacy skills was completed in 1994. NALS assessed the prose, document, and quantitative skills of adults of all ages

and educational levels. NALS analyzed the variations in skills across major subgroups in the

population as well as social and economic variables such as voting and earning. Although it

could not provide a direct link between classroom learning and the skills needed in the workforce, it nonetheless provided a good overview of the various levels of adult literacy across the

nation for a number of sub-groups. The design and size of the sample restricted the analysis to

the nation, although a few states funded additional sampling to allow them to assess literacy

within their state boundary.

Problem: Given the shortage of resources and the need to obtain information on the state of

learning across the nation, it is clear that alternative approaches to gathering information on the

attainment of student learning at the college level need to be identified. A recent publication by

the State Higher Education Executive Officers Association reports that all but two states have formal efforts focused upon the assessment of student learning? An ACT publication indicated that

many of the skills areas noted in the National Education goals are being assessed, although unevenly. Information on assessment of college student ”learning collected by states offer a potentially valuable source of information across the nation. However, the approach to assessment and

the way information is collected and analyzed differs across the states. Can available state-level

information provide an insight into the progress being made across the nation toward the achievement of National Education Goal 6.5? To this end, it is important to note that the question is not

whether the individual state level data can be summarized across states to provide a single national summary report. Rather, can the individual state efforts be used to develop a collctive view

of the progress being made to assure college graduates have the skills they need for admittance to

and success in the workforce once they leave college?

Planned Activity: A workshop is to be held, which will bring together directors of postsecondary assessment from each of the 50 states and selected territories. The meeting agenda will center around the following issues:

1.

2.

Identification of Skills Being Assessed. What knowledge or skills are being taught at the

state level and how do they relate to the skills identified in the national education goals or

other areas of national interest? What levels of achievement are expected? Are the skills being assessed at the secondary level consistent with the skills being assessed at the postsecondary school level?

Collection and Reporting of Skills. What is the frequency, reliability, consistency and compatibility of the information collected across the states? How are these skills being assessed

within institutions and across the state? At what grade level and how frequently? How are

the results being used?

41
3. Enhancement of Assessment at the State Level. What needs to be done to help states

broaden and enhance their assessment programs to provide for complete coverage of skills

in a more consistent and reliable manner?

4. The Role of NCES. What is the appropriate role of NCES in assisting states and institutions

gather and report the information and disseminate the results?

As the ECS study indicated, most states, as well as many private institutions, are engaged in assessing student learning in one form or another. It would be valuable, both to policy maker and

citizen alike, to gather this information and summarize the findings on a state by state basis. For

reporting purposes, it is suggested that each state be thought of as a case study. The value of this

approach is that individual states would be able to compare the results with comparable state efforts. This sharing of information can also be used to enhance the teaching/learning of these

skills within a state. Again the focus of this discussion will be on the sharing of information on

the development, implementation, and summary of assessment activities from a state perspective.

To enhance this process, it will be necessary to use the same terminology and reporting format.

Special consideration will be given to states which have successfully developed and implemented

an assessment approach or program.

Expected Outcomes: Two products are expected from this project. One is publication of an updated summary of current state postsecondary assessment_ activ~ties and results prepared by workshop participants from each state. The second will be a report which includes workshop

proceedings on how state assessment activities might be enhanced along with a discussion on the

potential use of assessment activities at the state level to support the national education goals. It

will also include other papers and/or reports prepared for the working conference.

Meeting Plans and Logistics: The meeting will begin with a morning session on the first day,

December 7, and end at 12:30 the second day, December 8. Most sessions will be in a large

group meeting although breakout rooms for 20 people each will be used the morning of the second day. It is expected that state representatives will attend along with a number of consultants

and others from government and selected associations for a total of 90 participants. Tmnsportation and housing will be provided for the state representatives and consultants. State participants

will be asked to prepare a summary of their state’s assessment activities. As noted, the workshop

proceedings are to be published. A meeting agenda, prepared papers, and other information will

be sent to participants prior to the meeting.

42
APPENDIX B

STATE BACKGROUND PAPERS

43
>> .

ALABAMA

Origins of Assessment Interest in the State

The state level assessment policy reported in 1990 is still in effect, although institutions no

longer report annually on its implementation. This policy was initiated by the Alabama Commission on Higher Education. The policy directs state higher education institutions to develop their

own assessment procedures and to report the results in their respective Institutional Annual Planning Statements. This reporting requirement has been discontinued. The policy, “Institutional Effectiveness and Assessment of Student Learning Outcomes,” was primarily for the purpose of

statewide improvement of undergraduate education. At this point, the institutional assessment

plans are voluntary and are developed by the institutions themselves.

Funding

No new or distinct funding was attached to the assessment policy. There are no designated state

funds for assessment purposes, and each institution funds such activities through regular appropriations.

Statewide Articulation and General Studies Committee

In 1994, the State Legislature passed Act 94-202, establishing-an Articulation and General Studies Committee under the auspices of the Alabama Commission on Higher Education. The Committee is to develop a statewide general studies curriculum and articulation agreement by 1998,

examine the need for a uniform course numbering system by 1999, and resolve problems in transferring credit earned at one institution to another institution. The Committee has met monthly and

is well involved in developing the statewide general studies curriculum. The completion of this

project will enhance the efficiency of the higher education system by assuring students that they

are taking appropriate coursework that will not have to be repeated. It will also provide a statelevel framework for the resolution of articulation issues. This is an important step in relation to

assessment of student attainment of knowledge and skills which are essential regardless of one’s

major or vocational interests in that we can have greater assurance that assessment will relate to

learning experiences common to students in all institutions.

Exploration of the Need for External Standards of Educational Performance at Appropriate Points in the Undergraduate Curriculum

During the 1990-95 planning cycle, the Alabama Commission on Higher Education initiated discussions to explore the possible development of a statewide rising junior examination. While consensus has not yet been reached on the need for the test, progress has been made, and this is a

planning objective being recommended for continuation into the 1995-2000 planning period. In

view of the work of the Articulation and General Studies Committee to develop a statewide common core curriculum, a rising junior exam could provide an objective determination of student attainment of essential knowledge and skills prior to beginning upper division studies.

Additionally, the staff of the Alabama Commission on Higher Education is looking at the need

and feasibility of a statewide testing program for college seniors in their last term of study before

graduation.

44
Institutional and Statewide Performance Indicators

The 1995 report of the Alabama Joint Legislative Committee on Finances and Budgets stated:

“The public is demanding accountability for the use of its money.” The Alabama Commission on

Higher Education is currently considering a recommendation that a state-level performance indicator system be put in place that will allow the state to garner some sense of the trend and health

of its higher education system and its progress toward achievement of long range goals. The recommendation, if adopted, will promote the demonstration of quality by establishing a cluster of

performance indicators that will be included in overall institutional and statewide assessment efforts.

45
>-

ARKANSAS

Entering Freshmen

All full-time first-time freshmen must present scores on either the ACT. SAT, or ASSET. Accordingly, institutions must place students in college-level classes or remedial programs. Students scoring below state standards in mathematics, reading, or composition must enroll in

remedial work in the area or areas of weakness.

Post-General Education Core (Rising Junior)

All baccalaureate degree-seeking students must take the Arkansas Assessment of General Education (AAGE), which consists of a composition test and four CAAP tests (mathematics, science

reasoning, writing, reading). Afler completing forty-five semester hours of coursework and before completing sixty, students must receive a valid set of scores on CAAP. Since the AAGE is

designed to provide data for the assessment of institutional performance, rather than performance

of the individual student, no student who receives a valid score is prevented from advancing to

the junior year.

Performance funding goes to those institutions that demonstrate success in teaching courses in

the thirty-five-hour State Minimum Core. CAAP covers the fields represented in the core: composition, mathematics, natural sciences, arts and humanities; and 3ocial sciences.

Exit

Arkansas provides funding for demonstrated performance of students who have completed

courses of study. The program is being phased in beginning with the fields of nursing and teacher

education.

46
CALIFORNIA

Origins/Interest/Authority for Postsecondary Student Assessment in California

While California has no state-level mandate for assessment in 1990, the Legislature passed and

the Governor signed the “Higher Education Assessment Act of 1990. ” The law states that the primary purpose of assessment which the Act encourages but does not mandate—is to “improve

teaching and learning as well as academic advising.” Because California has not mandated any

particular form of assessment that its public institutions must use, it has left assessment mechanisms to the discretion of the institutions. California’s Higher Education Assessment Act also declares the intent of the Legislature to monitor the performance of its public institutions in the

following five areas: (1) diversification of the student body, (2) improved transfer rates, (3) improved retention rates, (4) diversification of faculty and staff, and (5) enhanced student learning

and improvements in students’ knowledge, skills, and abilities.

Primary Purpose of the Assessment Program

While California does not have a state-mandated assessment program, each of its public institutions has some form of assessment activities. These activities focus primarily on entering students to determine the student’s level of English and math proficiency and to determine the

course level at which the student most appropriately should be placed.

Description of the Assessment Program

University of California: Entering students are assessed to determine their writing proficiency.

If they fail the English writing examination, they must enroll in Subject A (an introductory English course).

California State University: Entering students are assessed to determine if they possess proficiency in writing and mathematics. If they fail to score high enough on these examinations, they

must enroll in introductory courses. These courses typically do not award degree credit. For California residents, results of those examinations are transmitted back to the high school from which

the student graduated. In addition, at the State University, upper-division students are also assessed to determine that their writing skills are adequate for college graduation. A student cannot

graduate from the State University until they have passed this upper-division writing test or

earned a passing grade in an upper-division writing course.

California Community Colleges: Through the community college “matriculation program,” all

new students who do not yet possess a college degree are required to take a series of tests to determine their proficiency in reading, writing, and math. Based on the results of those tests, students

are counseled as to which courses would be most suitable for them, however, such counseling is

advisory only as students can enroll in any course offered by the community college.

Validity of the Assessment Instrument and Reliability of the Process

Unknown.

Data and Test Results Collected Across the State

Data from the above tests are not collected on a statewide basis. Each public institution can typically provide information on the number of students who took and passed each exam and, in the

case of the State University, report this information to the state’s high schools.

47
,.

.? .

Reporting and Approval Requirements for Assessment Initiatives

Institutions are not required to submit their assessment plans to the state for formal approval, although local community colleges must have their matriculation program assessment instruments

approved by the state-level Community College Chancellor’s Office.

Funding Available for Assessment Activities

Funding for the assessment activities carried out by the state’s public institutions are not earmarked specifically for that purpose, but rather are part of the state’s overall appropriation to the

institution. As such, the state does not know how much is being spent on assessment activities.

Findings from the Assessment Activities as They Relate to National Education Goal 6, Objective 5

California’s higher education assessment activities are not related to the National Education

Goals and hence one cannot readily determine if Goal 6, Objective 5 is being achieved;

Contact: Karl M. Engelbach

Senior Fiscal and Policy Analyst

California Postsecondary Education Commission

1303 J Street, Suite 500

Sacramento, California 95814 .

Phone: (916) 322-7331

Fax: (916) 327-4417

E-mail: engelbac@cpec .ca.gov

48
COLORADO

Origins/Interest/Authority of Postsecondary Student Assessment in Colorado

In 1985, the Colorado legislature passed a bill that, among other things, required institutions to

develop an accountability process to assess what students learned between entrance and graduation. The statute, mindful of institutional role and mission, allowed each institution to develop

its own plan based on the structure of the statute and the policies devised by the coordinating

board. Accountability was part of a large bill that restructured the role of the coordinating board

and, in many ways, was an afterthought primarily developed because of difficulties with the K12 system. It did not address any identified deficiencies of the higher education system.

Primary Purpose of the Assessment Program

The purpose of accountability in Colorado was for higher education institutions to demonstrate

public accountability and to increase the quality of undergraduate education. The legislation required that institutions develop their plans (including goals, objectives, and measures) with maximum public input and, later at the request of a prominent legislator, institutions were required to

have the public review assessment results. The institutions report annually to the coordinating

board (CCHE) and CCHE then develops a report for the Governor and the General Assembly. Individual institutions may develop shorter reports for public consumption, especially for legislators in their districts. Some institutions have contributed information that resulted in newspaper

articles. But in general the public accountability efforts in Colorado have had no impact.

To meet the charge of producing well-educated undergraduates, the institutions have been more

successful. Institutions developed goals, measures, and review results for all undergraduate degrees and general education. Those results are reported annually to the coordinating board. One

of the requirements of the coordinating board’s report is that institutions must discuss the

changes made as a result of assessment findings. The institutions have made changes in curriculum, teaching skills and support services as a result of assessment. The primary value of this activity is the internal focus on undergraduate education to improve its quality.

A Brief Description of the Assessment Program

The coordinating board developed a policy that required institutions to assess students in the following areas: 1) general education knowledge and skills, 2) skills and knowledge in the major/vocational area, 3) retention and graduation statistics, 4) alumni and student satisfaction, and 5)

after-graduation performance. Institutions developed a set of academic program goals that were

due in 1988. After discussions between coordinating board staff and institutional staff, goals

were accepted or modified. The next step required institutions to build their plans for assessing

students based on their approved goals. The plans were reviewed and evaluated by the National

Center for Higher Education Management Systems and again a negotiation process between institutions and the coordinating board occurred. Over a period of six months every institutional plan

was approved. After an institution received approval, it began implementing its plan. The focus

of all plans concomitant with the legislation is to assess graduates’ learning in programs; all plans

measure learning in programs, not learning of individual students. Two-year institutions monitor

the results of their basic skills programs, while all institutions measure skill attainment in the general education curricula. Community colleges monitor the skill and knowledge development of

graduates with vocational degrees and certificates, as do the four year institutions for students ob49
. .

taining baccalaureate degrees. There is no policy requirement to assess support services, but

many institutions do so and report that data (e.g., library, tutoring. advising, etc.). The policy requires an annual institutional report based on the assessment results of their programs, and focusing on improvements made as a result of assessment.

Means to Establish Validity of the Assessment Instrument and the Reliability of the Process

Faculty have great flexibility to design their own goals and measures. Although the coordinating

board may question some of the measures developed, in actuality there is little that it can do to

make faculty change their measures. Many institutions have institutional research staff available

to assist faculty in developing test questions, conduct surveys, and interpret data. It is ultimately

the faculty, however, that choose the measures and decide if they are valid.

Common Data or Test Results Collected Across the State

At this time, there are no common test results (primarily because there are no common tests required) available statewide. The coordinating board, with the support of key legislators and the

Governor, is developing a proposal to collect certain “performance indicators” from each institution to meet the need for public accountability. At this time, the process envisioned is all institutions will collect and report a common set of indicators in addition to a set of unique indicators

chosen to reflect the institution’s role and mission, A certain unique set of indicators will be reported only at the state level. Only one indicator asks for the results of licensure and certification

tests (although the Governor has suggested a test to be given to all graduates); and few address

learning outcomes, other than student and employer satisfaction measures.

Reporting or Approval Requirements of the Assessment Initiatives and by Whom

The coordinating board has the ultimate authority for approval of the goals, plans, and annual reports submitted by institutions, but the governing boards are also required to approve these documents before submittal to the coordinating board. Occasionally, a governing board will require

that an institution rewrite a report. The majority of the interaction on assessment is between the

institution and the coordinating board.

Funding Available for the Past Three Fiscal Years

The institutions must find assessment through their base budget allocations; no additional moneys have ever been provided to implement assessment activities. Before the legislation was enacted, a quality improvement program (in the early eighties) that allocated grants for improving

instruction supported some early assessment efforts. Colorado’s coordinating board has the dubious distinction of having the authority to decrease an institution’s budget by 2°/0 if the institution

does not implement an accountability program. Although the 2?40 provision provided leverage to

the coordinating board at the begi~ing of the process, it could never be implemented politically.

Brief Overview of Findings

Since each institution develops its own program, there are no overall statewide findings. In general, the community colleges have not been successful in measuring the impact (if any) of basic

skills courses. Instead some of their data shows that developmental courses may have no impact.

General education skills of writing and computing are measured by all institutions with essays

and math tests (usually). In general, institutions can show increases in writing skills from basic

freshman classes; however, that increase may not be maintained throughout the student’s career

50
if the student is not required to continue to write in classes. Many faculty have commented that

seniors do not write at the expected level. Math skills follow the same pattern. Some institutions

develop oral presentation assessments using speech classes. Knowledge gain from general education classes is sparse and not well-documented. The majority of institutions have a distributed

general education curriculum, and it is difficult to veri& that the curriculum produced any specific knowledge gain. Students will have taken a unique set of courses in a unique pattern of

course taking. It is very difficult to say that choosing 20 courses randomly out of many possibilities will always produce the outcomes desired in vaguely-written general education goals.

Faculty develop measures and review results of assessing the major/vocational skills with the

most enthusiasm. Measures include papers, presentations, student surveys, capstone courses and

tests. The majority of these measures test the skills and knowledge needed in the discipline/ vocational area, with few measures of the “soiler skills’ ‘-better gitizens, increased volunteerism, enhanced aesthetic appreciation, etc. This is the area where most employers and faculty find

students to be lacking the necessary skills. Employers agree that students (with the possible exception of those holding certificates) have the necessary technical skills to perform the job, but

do not have the “soft skills”: communication skills, customer service, etc.

Committees consisting of local employers and professionals working in the field review the results and comment on the goals, measures, results, and possible changes. Committees provide a

good review of technical skills, but they often don’t se~ meaWres or results on other skills.

Special Assessment Needs

Following from the answers in the previous questions, there are two areas where additional assessment is needed. The first need is “soft skills” and these should be defined by employers. Faculty too often emphasize skills needed in their faculty positions (e.g., individual research papers

and oral presentations as measures of communication skills) in their program measures that may

be less relevant outside of academe.

The second need is a methodology for assessing knowledge gain in general education to measure

if general education goals are being met. A set of goals measuring general education, acceptable

measures, and useful results are lacking at most of the institutions in Colorado. After dismissing

standardized tests such as the ACT Profile and CAAP early in the assessment effort, institutions

are beginning to use them again because institutions have not developed anything to replace them

that is reliable and valid. It is not clear if standardized tests are valid for all general education curricula, but the tests do produce reliable results that can be compared against national outcomes.

Contact: Stephanie Cunningham

Academic Officer

Colorado Commission on Higher Education

1300 Broadway, 2nd Floor

Denver, CO 80203

303-866-2723

303-860-9750 (fax)

steph@cche.state .CO .US

51
;>

GEORGIA

Origins/Authority of Higher Education Student Assessment in the State

Under the aegis of the Board of Regents’ Office, Office of Planning and Policy Analysis, an ad

hoc Assessment Advisory Committee was appointed in 1987. Committee membership consisted

of institutional presidents, vice presidents, institutional research and planning officers and system

office staff. In December, 1989, the Board of Regents approved a “Planning and Assessment”

policy statement that requires every institution in the University System to have a summary of

significant assessment results and associated improvement objectives along with action plans by

which improvements in effectiveness will be achieved. In addition, Board policy directs each institution to describe the process by which systematic assessment of institutional effectiveness is

conducted and the results of assessment are used to achieve institutional improvement. Policy

also mandates that each institution will link its major budget allocations and other major academic and administrative decisions to its assessment process. The policy also permits assessment

procedures to differ from institution to institution, as long as each program includes the assessment of basic academic skills at entry, general education, specific academic program areas, and

all academic and administrative support programs. The formal Board policy is supplemented by a

set of assessment “Resource Manual s,” which provide structure for a standard assessment model,

but provide great institutional flexibility in actual implementation.

The Primary Purpose of the Assessment Program

The primary purpose of the assessment program is improvement of the function assessed. Summaries of outcomes and expected results, descriptions of assessment methods, assessment results

and improvement plans for each institution are maintained in the Board of Regents Office. It is

the individual institution’s responsibility, and the responsibility of the faculty/staff of individual

programs to adopt and implement their respective approaches to assessment.

A Brief Description of the Assessment Program

Underprepared entering students in the University System of Georgia are assessed through the

Collegiate Placement Exam (CPE) or the COMPASS exam to determine their entering level of

preparation for college work. The University System also conducts a Regents’ Test of reading

and writing skills for rising juniors.

General education is assessed according to predetermined learning outcomes and expected results

in such areas as communications, mathematics and science, and cognitive reasoning. Each institution establishes its own set of general education learning outcomes and assessment methods.

The assessment of degree programs is the responsibility of each program’s faculty who define

learning outcomes, select and implement assessment methods and analyze results and plan the indicated program improvement.

All student learning outcomes are based on knowledge, skills, values, attitudes and/or behaviors

that students are expected to acquire from the respective program.

Non-academic outcomes, for all administrative and academic support functions, are defined by

the staff who perform those functions. It also is their responsibility to select or design and execute assessment methods and use the results to improve the effectiveness of the function assessed.

52
Validity of the Assessment Instrument and the Reliability of the Process

Consistent with the decentralized “ownership” of the assessment process in the University System, assessment methods are selected or developed and implemented by the faculty, as appropriate to the particular academic program being assessed. Validity and reliability are stressed in all

System assessment literature, as are the use of multiple methods and the appropriateness of both

qualitative and quantitative methods. The System Resource Manual on Assessment Methods offers examples and guidance in a wide variety of methodological approaches.

The reliability of assessment methods is determined, ultimately, by the accuracy of the insight

into program improvement that they provide. Program improvement will be evident in the greater

effectiveness in student attainment of learning outcomes that is revealed by subsequent assessment cycles.

Common, Statewide Data or Test Results

Other than the CPE for some entering students and the Regents’ Test for rising juniors, the former a placement test and the latter a graduation requirement, there are no common, statewide

data or test results prescribed or collected in the University System.

Reporting/Approval Requirements of the Assessment Process and by Whom Administered

Institutions are required to submit summaries of their assessment processes, including a description of the organizational structure and process and how it is administered; outcomes and descriptions of the assessment methods; examples of assessment data obtained by the process and

improvements made as a result (preferably with evidence of improved effectiveness). Approval is

not required, but the Board of Regents staff has responded to institutional submissions with detailed suggestions for assessment program improvement. Beginning in 1996, the critique of institutional assessment summaries will be performed by peer review teams consisting of institutional

personnel.

Available Funding for the Last Three Years

No special funding has been available to support outcomes assessment in the University System.

Presidents in the system receive a lump sum allocation from the Chancellor, and they are expected to fund assessment internally. There is no line item funding in Georgia.

A Brief Overview of Findings

Because the University System assessment program is still in the developmental stage, with implementation proceeding at each institution’s own pace, there is not a set of comparable findings

available for any specific area.

Special Assessment Needs in the State

The University System approach to assessment is open-ended, institutionally based, and not

linked directly with specific external goals.

53
> ---- .-

Contact: General Education— Dr. Cathie Mayes Hudson

Associate Vice Chancellor, Planning and Policy Analysis

Major Area Programs— Dr. Joseph J. Szutz

Assistant Vice Chancellor, Planning

Board of Regents—University System of Georgia

244 Washington Street, SW

Atlanta, GA 30334

(404) 656-2213

(404) 657=$130

chudson@mail.regents .peachnet.edu

jszutz@mail.regents. peaclmet.edu

54
-.

HAWAII

Origin/Interest/Authority of Student Assessment in the Public Higher Education Sector

In 1986. the State of Hawaii enacted Acts 320 and 321 granting the University of Hawaii additional administrative authority in the areas of warrants, budgeting, disbursing, expenditures, and

emergency bids. These measures were adopted for a three-year period and were commonly referred to as the “flexibility legislation. ”

As part of Acts 320 and 321, the Legislative Auditor was charged “to assess and evaluate” any

impact of the administrative changes set forth in the Acts “on the quality and effectiveness of instruction, organized research, public service, academic support, student services, and institutional

support program areas as applicable at the University . ...” Particular emphasis was to be given to

the impact of general fund transfer authority upon student education.

In 1987, the Legislature adopted resolutions requesting the University of Hawaii to develop and

implement an educational assessment program.

In 1989, Act 371 extended the administrative flexibility previously granted to the University of

Hawaii and the Department of Education. This law required the University to submit an annual

report to the Legislature on the progress of its educational assessment activities. This report was

to include the status of educational assessment programs witlih the University and indicate interrelationships between educational assessment activities conducted by the University and the Department of Education.

Within the University, Board of Regents policy and executive policy directing the University’s

educational assessment efforts were adopted in the 1989-90 time period.

The Primary Purpose of Assessment Programs

The purpose of the University of Hawaii’s assessment effort is provided in Executive Policy Section 111. A., page 2 of 9.

A Brief Description of the Assessment Program

The University of Hawaii assessment program is described in its annual reports on this subject.

Annual reports have been prepared since 1988.

As can be seen from the reports, no one assessment instrument is used across the system. The entire assessment process is quite decentralized.

Common Data or Test Results Collected Across the State

Again, no common test results are collected state-wide. Samples of outcomes from the various assessment activities are summarized on pages 16-21 of the Board of Regents’ Briefing on Educational Assessment, January 20, 1994. Also, the University system prepares an extensive array of

annual information reports on students and courses, surveys entering and graduating students,

and tracks student progress within and across the system.

55
-.

Reporting or Approval Requirements of the Assessment Initiatives and by Whom

Reporting requirements are summarized in the University’s Executive Policy, Section V.B.2. Specifically, each campus prepares an annual summary report on assessment activities for the Office

of the President. These reports are then compiled into a system report and shared with the Board

of Regents and Legislature as appropriate.

Funding Available for the Past Three Fiscal Years, by Program Area If Possible

This information is not available. Assessment activities are not identifiable as separate budget

lines at the campus or system level. The system office sets aside a small amount of money to support specific assessment projects, but this would not adequately reflect the total support of assessment activities.

A Brief Overview of Findings by Class Level %

This information is not available.

Special Assessment Needs Identified in the State Over and Above Those Identified in the

National Education Goals

Hawaii has given special attention to assessing the interrelationships between the University of

Hawaii and the Department of Education.

.-

Contact: Dr. Colleen O. Sathre

Vice President for Planning and Policy

University of Hawaii

2444 Dole Street, Bachman Hall 110

Honolulu, HI 96822,

Phone: 808/956-7075

FAX: 808/956-91 19

E-mail: csathre@mvax.its. hawaii.edu

56
?

ILLINOIS

Origins of Assessment Interest in the State

The Illinois Board of Higher Education adopted policies designed to improve undergraduate education, particularly student achievement, scholarship, and general education in September 1986.

The policies were updated in 1990 by the Board’s Committee on the Study of Undergraduate

Education.

Description of the Initiative

The policies on undergraduate education called for colleges and universities to define their objectives for general education and skill development within baccalaureate and baccalaureate-transfer

degree programs, to communicate these objectives to students? to assess individual student progress toward meeting these objectives and to conduct regular reviews of the undergraduate education experience. Colleges and universities were to collect data on the progress, retention, and

completion of all undergraduate students as a basis for regular review and improvement of the undergraduate curriculum and support services. The policies call for colleges and universities to use

multiple measures and to assess student progress at appropriate intervals.

At the state level, information systems were developed to support the Board’s policies by

1) providing feedback to high schools on the achievement of their graduates at public universities; 2) providing information to colleges and universities on the progress, persistence, and success of their students including students who transfer among community colleges and public

universities; and 3) providing public universities with information about the occupational placement and success of their baccalaureate graduates.

Individual institutions have undertaken a variety of assessment activities including writing examinations, team-graded common course final examinations, general education field examinations,

and senior-year capstone seminars or other experiences. Many institutions restructured their general education requirements and redefined expectations for students’ learning prior to developing

new assessment methods.

Primary Purpose

The purpose of the Board’s policies is the improvement of undergraduate education. Assessment

supports this objective by improving the chances for success for individual students and by providing information for examining the quality and effectiveness of programs and institutions.

Common Data or Test ResuIts

Two information systems, the Shared Enrollment and Graduation System and the Baccalaureate

Follow-up System, allow monitoring of statewide progress toward broad objectives for undergraduate education. The primary assessment activities are taking place on campuses where processes and techniques are selected or developed.

Reporting or Approval of Assessment Initiatives

Public universities and community colleges are required to submit annually reports on reviews of

undergraduate education. Assessment was the focus topic for reviews submitted in July 1993.

The results of assessment activities are also referenced in program reviews and in institutions’ Priorities, Quality, and Productivity reports.

57
. . >,.

Funding

No funding has been specifically targeted to assessment nor is funding linked to assessment results. During the last decade, high priority has been placed on undergraduate education in the

budget development process and new resources have been allocated to this priority in most years.

Contact: Kathleen K. Kelly

Deputy Director, Academic Affairs

Board of Higher Education

4 West Old Capitol Plaza

Springfield, Illinois 62701

217-782-3442

58
?

INDIANA

Origins/Interest/Authority for Postsecondary Student Assessment in Indiana

Indiana has no state-level postsecondary student assessment program. Authority for postsecondary student learning remains vested in the postsecondary institutions. There is no legislative

mandate for either state-level or institutionally based assessment. The Commission for Higher

Education is a coordinating board with no authority, barring legislative action, for introducing

postsecondary assessment.

Some institutionally-specific assessment initiatives are underway, none of which are comprehensive. Ivy Tech State College uses ACT’s ASSET assessment for student intake and placement

purposes. Vincennes University has introduced course-based performance assessment for some

courses and programs. Ball State University has implemented a writing assessment for all students. Indiana University at Bloomington is exploring performance assessment related to general

education.

What the state does have is a set of experiences that may have the effect of raising interest in the

postsecondary assessment of student learning. These include

●

●

Experience with K-12 assessment, and specifically with m~attempt to shift the original Indiana

Statewide Test of Educational Progress, or ISTEP toward a performance-based assessment program.

In 1992 new legislation called for changes in K- 12 testing, many of them based on the Oregon

model. Proposed changes included a shift to performance-based assessment (I-PAS S) at several grade levels including grade 12, and a high-stakes “Gateway” exam to be required for high

school graduation. In anticipation of these new assessments at both the Gateway and grade 12

levels, the public postsecondary institutions agreed to work with the State Board of Education

on standard setting and on connecting the new standards with revised admissions procedures.

The 1995 General Assembly revisited the assessment issue and decided to move forward more

cautiously. In the near future, K-12 assessment will include both a norm-referenced, multiple

choice component and a writing component. The initiative continues to evolve, At the postsecondary level, what remains is an interest, supported by the Commission, in encouraging institutions to pursue their own, independently designed, assessment initiatives.

A jointly adopted Board of Education/Commission for Higher Education plan for improving

the fit between high school and college.

In February, 1994, the two boards adopted a plan which, among other things, specified a high

school core curriculum for all students. Implementing “Indiana Core 40” has required the development of course-specific competencies in core subject areas. Identifying these competencies has engaged college faculty along with high school teachers in team efforts. From this

effort should follow university-based professional development workshops for high school

teachers that will address performance assessment at the classroom or building level.

59
>- .

●

●

●

A series of Department of Workforce Development initiatives related to tech prep and to the assessment of competencies acquired by students taking occupational programs. Much of this, especially the latter effort, is based upon the Oregon model.

A state-level pilot project (Indiana Performance Assessment) organized by the Commission for

Higher Education in 1991-92 and funded from non-state sources.

The project developed about ten experimental language arts and mathematics assessments that

were administered to high school and college students and also to adults seeking admission to

Ivy Tech State College. The project resulted in a major assessment initiative within the College of Arts and Sciences at Indiana University in Bloomington. The project director also consulted with faculty at Indiana’s two public two-year institutions, although that has not yet led

to any comprehensive effort to assess student learning at either institution.

A now-defunct effort to encourage individual institutions to develop measures for the assessment of student learning. Active through two biennia in the late 1980s, this effort was abandoned in 1989, leaving assessment tacitly recognized as an institutional responsibility.

Contact: Karen Rasmussen

Commission for Higher Education

101 West Ohio Street, Suite 550 - -

Indianapolis IN 46204

Phone 317-464-4400/Fax 317-464-4410

e-mail “karen@?chevax. che.state.in.us’’@ chevax@mrgate

60
1.

2.

3.

4,

5.

7.

8.

9.

10,

IOWA

Board of Regent policy adopted in 1991 requires every institution under the Board to undertake outcomes assessment for every program. This grew out of an interest in this office for

improving student learning.

Primary purpose is to improve student learning. Secondary purposes are to improve teaching and to provide an outcomes assessment component to on-going program review efforts.

The choice of assessment varies from program to program. The specific choices as to what

outcomes assessment will be undertaken is a faculty choice based on the nature of the academic discipline involved.

The validity is faculty determined, with assistance from the central administration.

No common data or test results are collected across the state. Each program reports on its

methods, procedures and use annually (although consideration is being given to a biennial

collection). This report is provided in summary fashion to the Board.

See above.

Assessment activities are budgeted through each institution.

It is not our purpose to assess “National Goals”. - -

See above.

Contact: Robert J. Barak

Deputy Executive Director

Board of Regents

Old Historical Building

E. 12 & Grand Ave.

Des Moines, IA 50319

515-281-3934 Phone

515-281-6420 FAX

E-mail: rbarak@iastate.edu

61
Origins of Assessment Interest in the State

The staff and members of the Kansas Board of Regents have been the primary initiators of assessment in the state by developing and approving a format in October 1988, Institutions are to create

their own assessment design and activities, however each design must assess basic skills, general

education and student learning in the major. Since 1988, the Council of Chief Academic Officers

and Council of Faculty Senate Presidents have led an effort to improve the assessment of the maj or.

Description of the Initiative

Based on the format approved by the Board of Regents, all public colleges and universities in the

state have created their own assessment plan based on institutional role and mission. The institutional plans were approved by the Board of Regents in January 1989. The plans have been implemented over a three-year cycle. While segments of institutional assessments are being phased in,

all plans are currently operational.

According to the format, the plans had to create and identify expectations for baccalaureate degree students in three areas: basic skills, general education, and the major field of study. The

plans had to further show how institutions will measure attainment of these expectations and use

assessment data to improve programs.

Primary Purpose

The primary purposes of assessment are to demonstrate publicly the effectiveness of higher education and to stimulate appropriate educational reforms. Assessment results are reported to the

Kansas Board of Regents.

Are Common Data or Test Results Collected Across the State?

Assessment is being conducted at the institutional level. There is no statewide mandate requiring

uniform testing and data collection.

Is Reporting or Approval Required of Assessment Initiatives?

A schedule for reporting development and implementation of assessment activities was included

in each institutional plan submitted to the Board of Regents. Institutions report on assessment of

basic skills and general education annually; assessment of the major is reported once every three

years.

Funding

Institutions are funding assessmen~ activities with current appropriations.

Comments

Interest in assessment is expected to increase as progress reports from the institutions are submitted to the Board. Performance funding is also being studied as another initiative to strengthen colleges and universities.

62
Contact: John F. Welsh, Ph.D.

Director of Academic Affairs

Kansas Board of Regents

700 SW Harrison Suite 1410

Topeka Kansas 66603-3760

913-296-3421 -phone

913-296-0983- fax

E-mail: John@KBOR.State. KS.US

.

63
-.

.

KENTUCKY

Origins of Assessment Interest in the State

Legislation was passed in 1992 (codified as KRS 164.095) which specified accountability indicators to be measured through a collaborative effort between the Kentucky Council on Higher Education (KCHE) and the public institutions of higher education. In addition, an objective of the

recently-approved Strategic Plan for Kentucky Higher Education 1996-2000 is for institutional

programs to “identify and evaluate students’ educational outcomes relative to program priorities

within institutional missions. ” Finally, a system of performance funding is under development as

part of an overall funding model for higher education. The framework for this system is based on

the goals and objectives of the Strategic Plan. Thus, current assessment activities are based on

the directives in the 1992 legislation and the planning/perfor~ance initiative.

Prior to formal legislation, the 1990 General Assembly funded the Accountability Enhancement

Program to be coordinated by the KCHE. This program awarded mini-grants to institutions for

projects related to student outcomes assessment.

Primary Purpose

The accountability reporting is a response to increased public demand for evidence of effectiveness in higher education. The accountability reports are submitted to the governor and legislature

each December. In addition, reports are sent to KCHE, university governing board members,

state agencies, faculty groups, non-profit education and economic development organizations,

public libraries, and numerous individuals. Accountability results are used in updating and monitoring implementation of the systemwide strategic plan, in the academic program review and approval processes and in fiscal policy development.

Institutions use accountability and other assessment results to (1) make program improvements,

(2) measure institutional effectiveness, (3) improve student support services, (4) monitor student

progress through studies of different populations, (5) improve public service programs, (6) enhance strategic planning and funding decisions, and (7) meet program and institutional accreditation requirements.

The S(ralegic Plan establishes system priorities. Performance funding is intended to recognize

progress in key areas of the plan.

Description of the Initiative

The accountability legislation directed the KCHE and the institutions to develop and implement a

systemwide accountability reporting process based on 14 specific indicators. The 14 indicators

addressed in these reports respond-to the following three questions:

. How satisfied are.. students, alumni, parents, clients, employers?

. How successful are.. education reform efforts, research and public service efforts, remedial efforts. students who take licensure exams?

64
-.

. 170w many.. programs are accredited, students are enrolled, students persist and graduate, students transfer from community colleges, degrees are awarded, student credit hours are generated, hours do faculty work, faculty hours are spent instructing and who teach the classes,

rooms are used, course sections are added or overrides approved to meet student demand, credits/semesters to earn a degree?

*

An accountability plan and detailed reporting procedures were developed in 1992-93. The 1993

reports established baseline data for each institution on each indicator. Upon reviewing baseline

data, institutional achievement goals for each indicator were approved by KCHE and reported in

the 1994 reports. Progress will be assessed after a five-year period ending in 1997, although progress toward achieving the goals is reported annually. All reporting is guided by a detailed Procedures Manual which establishes the definitions and reporting guidelines for each indicator. In

most cases, results are provided by level of student or progr~ by discipline, by race and gender,

and other appropriate breakdowns.

Each year separate accountability reports are prepared for each institution and distributed to interested parties throughout the Commonwealth. Institution-specific reports are developed to show

progress of a given institution compared to itself and to offer the most useful format for policymakers interested in the results. Individual reports are prepared for each of the eight universities,

each of the 14 community colleges, and a summary for the community college system. An overall system report presents aggregate data by type of institutio~ and provides a system perspective

of results over the reporting period.

The Strategic Plan identifies selected desired results that would reflect achievement of system

goals and objectives related to quality programs, educated citizenry, equal opportunities, economic development, quality of life, coordination and advocacy. In some cases, accountability reporting will help measure results particularly related to educational attainment. The framework

for the performance piece of the higher education funding model is based on the goals and objectives of the Strategic Plan.

Common Data Collected Across the State

Detailed accountability reporting procedures were developed for reporting consistency among

the institutions. These procedures include definitions for all relevant data elements and common

collection and reporting guidelines.

Flexibility has been built into the reporting process in recognition of differences in institutional

missions and planning objectives. Institutional reports measure each institution’s progress compared to its progress in previous years while using common data definitions across all institutions. A balance was also struck between commonality and individuality in the development of

goals. Three types of goals were established against which to assess progress on the accountability indicators:

● Unz~orm Targets. These goals establish a minimum standard of acceptable performance for indicators for which continuous improvement over time would not automatically be expected

(e.g., student scores on licensure exams).

65
.-

●

●

;>. .

Ins[jtutional Targets. These goals set specific numerical targets when the direction of progress

was the same across institutions and institutional gains could be measured easily using a standard methodology (e.g., entry-level course pass rates for remediated students).

Evidence of Effective Strategic Planning. For indicators where institutions may differ in their

assessment of what constitutes progress, guidelines require the institutions to place results in

the context of their planning objectives, provide evidence of relevant activities and discuss institutional progress toward meeting the objectives (e.g., enrollment data where institutions may

strive to increase or to reduce their current enrollments based on their planning objectives).

The Strategic Plan objective to identify and evaluate student educational outcomes allows each

institution to determine the most appropriate assessment method(s) consistent with institutional

missions which have been approved by KCHE.

The performance funding measures make use of selected accountability indicators. Institutional

missions drive the definition of several measures whereby different types of evidence are accepted in the context of individual missions.

Reporting or Approval Requirements

The accountability plan and the 1997 goals were approved by the KCHE. Annual reporting of accountability results is required. Biennial status reports will be requiFed on the results of efforts to

meet the Srrategic P/an objectives approved by KCHE. The performance funding system will require approval by KCHE, along with routine reporting on results in preparation for making funding allocations.

Although KCHE approval is required in the areas of accountability, strategic planning and performance funding, all of these activities are developed in close coordination with the institutions.

Work groups chaired by KCHE staff and composed of institutional representatives develop final

draft materials for consideration by KCHE.

Funding

Each university and the community college system receives state funding to support accountability reporting. The KCHE also receives an appropriation for administering the project. Approximately $230,000 is appropriated to the Council and a total of $1.2 million is appropriated to

the eight universities each year to support the implementation of the accountability program.

A portion of the new money for the community colleges and universities will be allocated based

on performance on specific measures effective with the 1996-98 appropriations.

Overview of Findings

The 1994 Kentucky system accountability report is attached. The 1995 report will be available in

December.

Comments

Three interrelated initiatives support Kentucky higher education’s efforts to demonstrate its effectiveness: the annual accountability reporting process, the KCHE’S Strategic Plan for Kentucky

Higher Education including a specific objective to evaluate students’ educational outcomes, and

the revised Funding Model including a system of performance funding. The goals and objectives

contained in the recently-revised Strategic Plan comprise the framework for selecting the per66
-.

formance funding measures. The desired benefits resulting from the Strategic Plan and the performance funding measures consider but are not limited to the available accountability indicators.

Contact: Joanne Lang

Deputy Executive Director for Planning and Policy Studies

Kentucky Council on Higher Education

1024 Capital Center Drive. Suite 320

Frankfort, KY 40601-8204

Phone: 502/573-1555

Fax: 502/573-1535

E-mail: JLANG’YoCHE@MSMAIL. STATE. KY.US

.*

67
,.

MAINE

Origins of Assessment Interest in the State

The Chancellor of the University of Maine System was the primary initiator of assessment activities in the State. Assessment was made a goal and a priority when Chancellor Robert L. Woodbury took office in 1986.

Shortly after that date, Chancellor Woodbury asked the Vice Chancellor for Academic Affairs to

convene a Systemwide Assessment Committee to begin to focus institutional efforts on improving teaching and learning. In a planning process involving study of assessment efforts in other

states, a group of individuals representing all seven campuses in the system attended the AAHE

Assessment Forurn in Denver in 1987. Within the state, the assessment committee also conducted a conference on assessment with a broad range of attendance from all the campuses. The

conference was held in November 1988 and was devoted entirely to student outcomes assessment. Out of these planning efforts, four major principles emerged to guide the development of

assessment initiatives. The principles are:

Assessment is much more than giving standardized tests.

Each campus in the System has a unique mission so therefore the best assessment program for

that campus will be one developed to reflect the mission. -

The most effective assessment is linked to teaching and learning in ways that will improve

teaching and learning.

The most effective assessment is supported and developed by faculty and viewed by faculty as

a way to improve the learning process.

The planning efforts in regard to assessment were underwritten in part by a pool of funds made

available for the years 1988, 1989, and 1990 from the Vice Chancellor’s budget. The funds covered the costs of four pilot assessment projects at the University of Maine at Farmington, a summer project in computer science at the University of Maine, and planning efforts at the

University of Maine at Fort Kent.

With the completion of the planning phase, the campuses have begun to implement their individual assessment procedures. While each institution has created its own process or processes, they

share elements in common to assessment plans that are operational elsewhere in the country. The

activities that the Maine campuses have chosen include student portfolios, capstone examinations in the major, surveys of students and alumni, focus groups, and the development of a Learning Center on one campus as a focus for assessment efforts. Indeed, the assessment activities are

program specific within a campus. -

Primary Purpose

The primary purpose of the assessment initiative in Maine is the statewide improvement of undergraduate education.

Description of Assessment Program

There is no statewide assessment program. Assessment activities in Maine are conducted by the

seven institutions. Examples are:

68
University of Maine. Honors Program capstone experience of researching and writing a senior

thesis or project.

University of Maine at Farmington. Teacher education programs engaged in developing performance based assessment. (b’What does a beginning teacher need to know and be able to

do?”)

University of Maine at Machias. Behavioral Science students participate in an exit interview

just prior to completion of the program.

University of Southern Maine. Counselor education program surveyed employers of graduates

to ascertain how well prepared employers feel these individuals are for their positions.

Reporting or Approval of Assessment Initiatives

Assessment has been incorporated into the planning goals for the University of Maine System

which are approved by the Board of Trustees. Assessment activities are reported regularly to the

Board.

Funding

The seven institutions of the University of Maine System receive a general annual appropriation

from which they are expected to fund their assessment activities.

.

Contact: Nancy MacKnight

Vice Chancellor for Academic Affairs

University of Maine System

107 Maine Avenue

Bangor, Maine 04401

Phone 207-973-3232

FAX 207-973-3296

E-mail: nancym(ijmaine.maine.edu

69
MARYLAND

Origins of Student Assessment in the State

The Higher Education Reorganization Act, passed by the Maryland General Assembly in 1988,

established an accountability process for public colleges and universities in the state. According

to the statute, each public institution must submit, and the Maryland Higher Education Commission must review and approve, a student assessment plan ($11-304) and annual progress reports

($11-306).

Primary Purpose of the Assessment Program

The goal of the assessment program is to encourage public colleges and universities, and particularity their faculty, to improve student learning, instructional effectiveness, and curriculum.

Description of the Assessment Program

In 1991, the Maryland Higher Education Commission required each two- and four-year public

campus to develop a plan for the assessment of undergraduate student learning outcomes. The

Commission approved these plans and has received progress reports from the campuses for

1992, 1993 and 1994. Each report 1) provided data on common and institution-specific indicators of student performance, 2) analyzed the significance-of the data to student learning outcomes, and 3) discussed the implications of the assessment program for innovations and changes

at the campus.

Means Used to Establish the Reliability of the Process

The Commission has issued reporting guidelines annually for the student assessment reports. To

enhance reliability, the guidelines have remained relatively consistent each year. However, slight

revisions have been made to eliminate indicators, such as trends in grade point averages, which

were found to be of limited use in assessing student learning outcomes.

Common Data or Test Results Collected

The reporting guidelines require a discussion of eight common indicators of student learning:

1.

2.

3.

4.

5.

6.

7.

8.

Effectiveness of general education programs

Student retention and graduation rates for all campuses and transfer patterns for community

colleges

Student evaluation of teaching

Admission of undergraduates to post baccalaureate study

Academic performance of co~unity college students who transfer to a baccalaureate program

Student performance on licensing, certification, and graduate admission examinations

Employment rates of graduates

Perceptions of graduates about the quality of their educational experience, as revealed in follow-up surveys

70
Campuses are asked to examine trend data based on these and optional institution-specific indicators and to explain the significance of their findings to the enhancement of student learning outcomes, particularly in relation to the mission of the institution. The campuses also are asked to

discuss the impact of the findings on institutional policies, services and educational practices related to student learning outcomes. This can include areas such as course content and prerequisites, teaching methods, entrance requirements, and student services, as well as the extent to

which the assessment effort has had specific educational benefits for students.

Reporting Requirements

The Commission staff reviews these reports to determine whether assessment findings resulted

in actions designed to enhance the instructional process. The Commission has reported to the

General Assembly on the progress which institutions are making towards the improvement of

student learning outcomes. This year, letters of appreciation and congratulation were sent by the

Commission’s chief executive officer to those institutions which demonstrated the impact of assessment on teaching and learning.

Funding Available

The accountability process was part of a reorganization effort that included significant base funding increases for higher education. These increases are expected to cover the costs of developing

and implementing the assessment plan and progress reports. -

Overview of Findings

It is evident from the reports that the assessment process is enhancing teaching and learning. Several institutions indicated that their assessment activities have resulted in the adoption of new

policies and procedures to improve learning outcomes for all students and have had an impact on

resource allocation. The reports indicate that most of the campuses are actively engaged in the

following:

Reviewing general education courses. Several campuses have instituted strategies to

strengthen the core curriculum.

Examining programs and curricula. Program reviews have resulted in enhancements to programs in several instances.

Sharing assessment findings with the college community. Most institutions have established

committees to oversee and strengthen assessment efforts. This approach has resulted in wider

dissemination of data to support assessment.

Reviewing student support services. Many institutions are implementing policy and procedure

changes as a result of student learning outcomes assessment.

Reviewing policies on placement testing. Community colleges seem to place more emphasis

on placement testing as it relates to student learning outcomes assessment. Several colleges reported that placement policies and procedures need to be revised in order to insure that students requiring developmental education are served.

71
-.

:>. ,

The document provides summaries of the student learning assessment activities of each public

campus in the state along with a staff analysis.

Other Comments

The Commission’s chief executive officer appointed a work group representing all segments of

higher education in the state and including faculty to develop performance accountability standards to measure campus effectiveness. This action resulted from legislative interest in the creation of specific accountability benchmarks and indicators that can help to measure how

efficiently public funds invested in higher education are being used. The work group will issue

its report this fall, and its efforts may bring changes in the accountability process in the state, including the way in which student learning outcomes assessment is conducted.

Contact: Michael J. Keller

Director of Policy Analysis and Research

Maryland Higher Education Commission

16 Francis Street

Annapolis, MD 21401-1781

PHONE: (41 O) 974-2971

FAX: (41 O) 974-5994

E-MAIL: MHECPO!MKELLER@MDHEC.A~TMAIL.COM

72
-.

MINNESOTA

Origins/Authority

The state legislature, with leadership of the Higher Education Coordinating Board, was the chief

initiator of assessment activities in the state with the passage in 1987 of a bill establishing a Task

Force on Post-Secondary Quality Assessment. The task force was directed to study the objectives of assessment and how it can be used to improve postsecondary education. The task force

was also charged with establishing a pilot assessment program within each of the public postsec -

ondary systems in the state.

In addition to funding the six pilot programs, the 1989 legislature re-authorized the task force

through June 1991 (it was scheduled to sunset in June 1989). The six pilot projects were funded

on a two-year cycle which continued through 1991.

Since 1991, there has been no statewide/state directed or mandated assessment of student learning in Minnesota. In that year, sweeping governance changes were enacted requiring the technical college, community college, and state university systems to be merged into one Minnesota

State College and University System by July 1995. Beginning in 1993, the Higher Education Coordinating Board came under attack in the legislature and in the 1995 legislative session, the

Higher Education Coordinating Board was abolished. A smaller Higher Education Services Office (HESO) was created in its place, primarily to distribute financial aid, collect student enrollment, financial aid, and other data, regulate private schools and colleges, lead the state

telecommunications initiative, and administer statewide and federal programs. Other activities,

including student assessment were left to the individual systems.

Minnesota policy-makers have not viewed establishing student assessment at the state level as a

priority initiative. In addition to governance issues, policy-makers have been more concerned

with postsecondary financing policies, providing remedial education for recent high school

graduates, time to completion, and establishing system performance measures. Among the performance measures legislated in 1995 were retention, transfer and graduation rates; job placement; minority enrollment and retention; and increased credits generated by telecommunicated

courses. Measures of student learning were not included for either the Minnesota State College

and University System or the University of Minnesota.

Contact: Dr. Leslie K. Mercer

Director, Data & Programs

Minnesota Higher Education Services Office

400 Capitol Square Building

550 Cedar Street .

St. Paul, MN 55101

(612) 296-6869 (phone)

(612) 297-8880 (fax)

E-mail: mercer@heso.state. mn.us

73
>..

.-

MISSOURI

Origins of Assessment Interest in the State

By the mid- 1980s Missouri was actively involved in discussions about state-level perspectives

on assessment, and Northeast Missouri State University was establishing a nationai reputation

for its value-added assessment progra,m. In 1986, with strong backing from the governor, state

educational leaders challenged all public institutions to establish assessment programs which

would improve student academic performance.

Description of the Initiative

Missouri’s assessment initiative, while mandating assessment at the state level, was not legislated. Consistent with its tradition of local autonomy, Missouri has used a permissive approach

that requires institutions to develop programs tailored to institutional missions, but does not mandate specific instruments or measurements. Beginning in 1987, institutions provided annual reports to the Coordinating Board about campus-based assessment programs.

As demands for accountability increased, new goals for higher education, including the reporting

of student outcomes data, were adopted by the Coordinating Board and agreed to by each campus. Missouri also expanded the data collection efforts of its Student Achievement Study, which

tracks student progress in the public sector from high sch601 through college graduation. Data on

performance indicators are collected on a regular basis and shared with all institutions, and with

the Coordinating Board; this information has replaced the annual reports about assessment activities.

In early 1991, the assessment coordinators at the public four-year institutions began a grass-roots

organization, the Missouri Assessment Consortium (MAC). This group, which has established a

strong network for communication, has chosen to remain informal. Principles for assessment, to

be used by each institution were adopted and promulgated by MAC.

In searching for new ways to directly influence campus behavior, Missouri has also engaged in a

performance funding initiative, Funding for Results (FFR), which seeks to link funding policies

with assessment and accountability initiatives. A subset of the performance indicators, selected

and refined with input from the institutions, has since 1991 been used as the basis for a performance funding component in the proposed budgets of Missouri’s public institutions of higher education.

With help from a grant awarded to the Coordinating Board by the Fund for the Improvement of

Postsecondary Education (FIPSE) in 1994, Missouri is refining and expanding its performance

funding program to emphasize the importance of teaching/learning issues. Missouri’s performance finding program rewards institutions at two levels. The first level addresses the contribution of institutions to the achievement of statewide goals, e.g., enhanced performance in general

education and in the major. A second level involves rewards for designing and implementing

mission-specific campus-level performance funding that is data-driven and emphasizes improvements in teaching and learning. Institutions are working on a variety of projects that require systematic assessment of student performance.

74
Primary Purpose

The primary purpose of Missouri’s public policies on assessment is the statewide improvement

of undergraduate education. A small portion of Missouri’s assessment initiatives involve graduate education. In addition, Missouri intends to use assessment activities to demonstrate responsiveness to demands for greater accountability.

Are Common Data or Test Results Collected Across the State?

Data on the broad set of performance indicators referenced above are collected annually. These

include productivity, efficiency, and quality measures. Each campus also gathers data in support

of campus-based assessment programs.

Is Reporting or Approval Required of Assessment Initiatives?

Institutions report their data on all state-level performance indicators annually, but the selection

of specific assessment instruments is still based on institutional priorities. The individual performance funding models of the institutions are evaluated by a peer group process using specified criteria, and subject to Coordinating Board approval. Institutions will be required to submit

annual reports including evaluative data to the Coordinating Board concerning progress on implementing a campus-based performance funding program.

Funding .

While rewards for doing assessment were included in the initial performance funding program established in Missouri, each year the amount of money recommended for simply doing assessment has been reduced. In general, however, institutions have been expected to fund assessment

efforts from a reallocation of current resources and appropriations. Currently, discussions are taking place concerning the costs of doing assessment and the extent to which performance funding

should continue to reward institutions for engaging in assessment activities.

As Missouri’s performance funding initiative matures, there has been a gradual shift to incorporate quality markers that can be attached to more traditional productivity measures. Overall, the

portion of Missouri’s higher education budget recommendation based on performance funding is

set at the margins. In FY 1993, 1 percent of the core operating budget was recommended. Actual

funding levels for FY 1996 were .9 percent for public two-year institutions and 1.2 percent for

public four-year institutions. Since this money is built into the core operating budget, institutions

have complete flexibility as to how they spend this money. For FY 1997, performance funding

recommendations represented 2.3 percent of the public four-year core operating budget and 2.9

percent of the public two-year core operating budget. These recommendations are under review

by the Governor and the General Assembly.

Comments

Institutions are currently implementing pilot performance funding projects, and developing plans

for fully-developed programs to be in place within the next year. These Funding for Results programs are driven by systematic assessment.

75
t

Contact: Robert Stein

Senior Associate

Missouri Coordinating Board for Higher Education

3515 Amazonas

.Jefferson City,MO65109

314-751-2361

76
>

MONTANA

Origins/Interest/Authority of Postsecondary Student Assessment in the State

The postsecondary institutions in Montana are developing assessment programs in response to

the standards established by the Northwest Association of Schools and Colleges, and each college and university is developing a program consistent with its own mission. Montana has no

mandated statewide postsecondary assessment initiative. The Commissioner’s Office will probably collect various indicators of academic progress for internal use and to prepare presentations

for state legislators.

Primary Purpose of the Assessment Program

Montana has no statewide assessment program.

Brief Description of the Assessment Program

Montana has no statewide assessment program.

Means Used to Establish the Validity of the Assessment Instrument

Montana does not have a common instrument used throughout the state.

Common Data or Test Results Collected Across the StateAt present, enrollment reports in a common format are the only data collected by the state office

from all institutions.

Reporting or Approval Requirements of the Assessment Initiatives and by Whom

Not applicable.

Funding Available for the Last Three Fiscal Years, by Program Area if Possible

Not applicable.

A Brief Overview of Findings by Class Level

Not applicable.

Special Assessment Needs Identified in the State

None.

Contact: Richard Crofts

Deputy Commissioner for Academic Affairs

Montana Systems of Higher Education

2500 Broadway -

Helena, MT 59620

(406)444-6570

rcrofis(ij),oche. oche.montana.edu

77
,-

NEBRASKA

Origins/Interest/Authority of Postsecondary Student Assessment in the State

There is currently no statewide mandate for student learning assessment in this state. Institutions

that are accredited by North Central have developed outcomes assessment plans and are in the

process of implementing those plans. The Coordinating Commission requests information about

student learning outcomes in its review of existing instructional programs. It also asks for outcomes assessment plans in proposals submitted by public institutions for establishment of new

academic programs.

Purpose of the Assessment Program

There is no statewide student learning assessment for Nebraska. However, one of the purposes of

seeking assessment data in the review of new and existing programs is to obtain evidence about

program quality. The data are used to enlighten decisions about the viability of new programs

that are proposed and about the continuation or discontinuation of existing programs.

Description of Assessment Program

There is no state-level assessment program,

Validity and Reliability

Not relevant.

Common Data

Not relevant.

Reporting or Approval Requirements

The only statewide assessment initiatives in Nebraska are under the responsibility of the Coordinating Commission for Postsecondary Education. The Coordinating Commission has expressed

support for the outcomes assessment requirements of North Central.

Funding

Not Applicable.

Overview of Findings

Not Applicable.

Special Assessment Needs

No special assessment needs have been identified.

Contact: Dr. Odus V. Elliott, Academic Officer

Coordinating Commission for Postsecondary Education

P. O. Box 95005

Lincoln, Nebraska 68509-5005

(402) 471-0023 (phone)

(402) 471-2886 (fax)

oelliott@ccpe.state. ne.us

78
. .

NEVADA

Origins of Assessment Interest in Nevada

The Board of Regents of the University and Community College System of Nevada adopted an

assessment policy in 1989 that requires each campus to develop a plan of regular student educational assessment. Initial plans were submitted to the Board in 1990.

Description of the Initiative

According to the policy, each campus is to assume responsibility for developing the assessment

processes and procedures to be used. Plans are to be based upon campus mission and involve

multiple assessment approaches. The policy also provides that plans are to reflect the mix of programs and types of students at each school. While assessment approaches vary among institutions, the universities and community colleges are to work together to develop common

approaches, where appropriate

The Primary Purpose of the Assessment Program

The primary purpose of assessment in the state is to stimulate curricular action and improvement

at the institutional level.

Brief Description of the Assessment Program -

Each campus has adopted a schedule of regular activities including a variety of surveys and interviews which are conducted of students, graduates, faculty, employers, community members, and

others. Results of college placement, graduate admission tests, and state and national licensure

examinations are also being used to provide feedback on instructional programs, college services

and employer satisfaction. Various class levels are utilized, depending upon the objectives.

Progress is underway at the universities to assess the core curriculum. At one campus, in the

mathematics area, a longitudinal study of students in the core curriculum math classes is underway. Writing and critical thinking assessment continues, and systematic examinations of student

learning in other areas of the general education curriculum are planned. A collaborative approach

between a community college and university tracks and assesses transfer students through a writing portfolio assessment program.

The results of the academic program review process, which is undertaken by each campus and reported to the Board of Regents annually, are also being utilized to improve and enhance programs.

The Means Used to Establish Validity of the Assessment Instrument and the Reliability of

the Process

Each campus has developed a variety of survey instruments which attempt to link classroom and

learning and the assessment methods. Longitudinal studies, systematic examinations of student

learning in the core curriculum area, and assessment in the major areas include test scores, interviews and focus groups. Two community colleges have initiated measurable student outcomes

though Competency-Based Instruction programs,

79
d

Common Data or Test Results Collected Across the State

The Board of Regents policy provides that the Chancellor’s Office, with the campuses, will develop appropriate measures of student persistence and performance, and will collect and monitor

these data on a statewide basis. However, due to a lack of statewide resources, this is not being

done at the present time.

Reporting or Approval Requirements of the Assessment Initiatives

Since the initial reports on campus assessment efforts were submitted to the Board of Regents in

1990, the presidents of each campuses report, on a biennial basis, the results of their continuing

assessment activities.

Both the community colleges and the universities also address assessment requirements of their

accrediting association (Northwest) during regular self-study and visitation activities.

Funding Available for the Past Three Years

Campuses fund assessment efforts with existing resources and current appropriations. No special

funding particularly for assessment has been granted at the state level, and information is not

available on individual campus expenditures by program area.

Brief of Overview of Findings

.

The creation of extensive and rigorous new core curriculum requirements for all students at one

institution, and plans to review the general education curriculum at a second, will call for a comprehensive examination of the impact of these changes in the future. Plans for the establishment

of systematic examinations to evaluate students’ abilities in English, mathematics and critical

thinking are underway. In addition, portfolio projects in specific majors related to writing and

critical thinking skills are in progress at one university.

Special Assessment Needs

The multiple assessment approaches are based upon individual campus mission and goals. Due

to limited resources and the need for campuses to focus on their unique mix of programs, special

statewide assessment needs have not been identified. However, we are in the process of identifying potential systemwide institutional performance indicators for implementation over the next

two-year period.

80
>

NEW JERSEY

New Jersey does not have a state-level collegiate assessment program at the present time. More

specifically, the previously-operational state-level Basic Skills Assessment Program (BSAP) and

the College Outcomes Evaluation Program (COEP) that did employ common assessment instruments no longer exist. Moreover, state-level coordination of the higher education system has

been restructured, with the Board and Department of Higher Education (which administered

BSAP and COEP) having been replaced by the Commission on Higher Education and the Presidents’ Council.

However, assessment and outcomes continue to be essential features of the New Jersey higher

education system. There is now in place a set of guidelines for basic skills testing and remediation, and another for institutional accountability reporting. While the latter does not currently call

for learning outcomes to reporting, the guidelines in both areas are continuing to evolve along

with the new coordinating structure.

Contact: Philip L. Beardsley

Director of Research

New Jersey Commission on Higher Education

CN-542 .

Trenton, NJ 08625-0542

(609) 292-4319

81
r’

NEW MEXICO

Authority for and Origins of Interest in Assessment

The New Mexico Commission on Higher Education is a gubernatorially -appointed board authorized by state statute to plan and coordinate postsecondary education. Among other duties, the

Commission prepares recommendations for state funding of public postsecondary institutions.

. . Although the Commission has historically considered assessment the prerogative of individual

institutions, two recent statutory amendments have increased the Commission’s authority in this

arena. A 1990 law requires the Commission to submit an annual report card to the governor and

legislature, including “the results of [each] institution’s learner outcomes assessment program

and steps taken to improve learner outcomes. ” A 1995 law authorizes the Commission to consider “educational outcomes assessments” when recommending funding for faculty salary adjustments.

New Mexico has no statewide program for the assessment of college student learning. When the

first state-mandated report card was prepared in 1990, the Commission found that institutional assessment programs varied widely; most institutions had little or no institution-wide assessment

activities. For these reasons, and because a number of other, process measures dictated by the report card law have proven difficult to obtain in a reliable manner, the report card requirement has

been fi-ustrated. The Commission has therefore launched an eff%rt to strengthen the capacity of

institutions to measure student learning and to gather and use other measures of institutional performance.

Current State Strategy and Action

The Commission’s current philosophy is that-at least for the next few years—the state’s two

dozen postsecondary campuses should each develop assessment programs tailored to their own

missions and circumstances rather than being required to participate in any central, common assessment system. This belief was reinforced by an analysis of institutional plans for expanding

outcomes assessment, submitted to the Commission in 1993. The analysis confirmed the disparate state of activity and capacity to perform assessments across the campuses. Because the

state’s regional accrediting body, the North Central Association of Colleges and Schools, was at

the same time expanding its accreditation requirements to include assessment of student academic achievement, the Commission chose to lend its support to that effort rather than develop a

redundant or competing mandate.

In 1994, with financial support from the state legislature, the Commission awarded incentive

funding to support two projects aimed at expanding assessment activities on campuses. One, conduct of a statewide assessment conference for faculty and staff, was initiated in Spring 1995 and

will be continued in Spring 1996. Two spin-offs from this project are a statewide organization of

campus assessment directors and initiation of a statewide newsletter on assessment. The state’s financial doldrums may limit the use of such incentives in the near future.

The Commission has recently reactivated an improved statewide, unit record data base. The initial products of the data base are statewide student tracking studies, reporting rates of student

persistence, transfer, and completion of degree programs. This information will be incorporated

in the Commission’s annual report, The Condition of Higher Education in New Mexico.

82
In 1996, the Commission intends to organize advisory groups of institutional researchers and academic officers to identify additional measures of institutional performance that might be collected and published on a statewide basis.

Summary

Although its legislature has recently strengthened state-level authority to conduct outcomes assessments, New Mexico has chosen to encourage institutional assessment activities that reflect

the diversity of its postsecondary institutions. Outcomes measures such as graduation rates are

being compiled and reported at the state level, but more direct measurement of student learning

is currently left to the discretion of individual institutions.

Contact: Dr. Bill Simpson

Deputy Director for Educational Programs

New Mexico Commission on Higher Education

1068 Cerrillos Road

Santa Fe, NM 87501

Telephone: 505/827-7383

fax: 505/827-7392

e-mail: bsimpson@unm.edu

83
d

NEW YORK

Origins of Assessment Interest in the State

For decades, institution-based assessment has been mandated by regulation of the New York

State Education Department.

Description of the Initiative

<

The regulatory mandate reads as follows:

To be registered, each curriculum shall . . show evidence of careful planning. Institutional

goals and the objectives of each curriculum and of all courses shall be clearly defined in writing, and a reviewing system shall be devised to estimate the success of students and faculty in

achieving such goals and objectives. The content and duration of curricula shall be designed to

implement their purposes.

For each curriculum the institution shall designate a body of faculty who, with the academic

officers of the institution, shall be responsible for setting curricular objectives, for determining

the means by which achievement of objectives is measured, for evaluating the achievement of

curricular objectives and for providing academic advice to students. The faculty shall be sufficient in number to assure breadth and depth of instruction qnd the proper discharge of all other

faculty responsibilities. The ratio of faculty to students in each course shall be sufficient to as-

-Wre effective instruction.

The regulatory mandate by the State Education Department affects all public, postsecondary institutions in the two major systems—the State University of New York and the City University

of New Yorkand all independent colleges and universities, as well as all the proprietary, degreegranting institutions in the state. In addition, CUNY-which includes 10 four-year colleges and

seven community colleges—has required a basic skills test of all entering students since September 1978. The test is for placement purposes only, and does not limit access. However, students

who fail the basic skills test upon entry must pass it prior to their junior year.

In 1994 the Board of Regents urged all institutions to develop their own indicators of performance, to measure that performance, and to publish the results. The Board also directed Department staff to work with the higher education sectors in the State to develop indicators of the

health for the entire system of 249 public, independent, and proprietary degree-granting institutions. That work continues in preparation for the next Statewide Plan for Higher Education.

Primary Purpose

The primary purpose of institutional assessment in New York is the statewide improvement of

undergraduate education.

Are Common Data or Test Results Collected Across the State?

The regulatory mandate of the State Education Department is designed to foster assessment efforts within the institution. It is not designed to collect data for purposes of comparisons among

the postsecondary institutions.

Each Statewide Plan and interim Progress Report provides trend data by sector on enrollment, degrees granted, remedial and developmental study and other indicators of higher education per84
formance. Graduation rate data is collected for all degree-granting institutions using a cohort survival methodology. Completion rate information is collected for all non-degree proprietary

schools and placement information by program is collected for all VATEA eligible institutions

and for all non-degree proprietary colleges.

Is Reporting or Approval Required of Assessment Initiatives?

Results of assessment based upon the regulatory mandate are used primarily in program review

and approval processes conducted by the Department of Education.

Funding

Institutions do not receive any specific or targeted funding for assessment. These activities are

funded through the general institutional appropriation.

Comments

Assessment is expected to increase in importance in New York. The periodic reviews of institutions by the Education Department are beginning to focus more on “outcomes” as opposed to “inputs. ”

Contacts:

Barbara W. Flynn -Thomas J. McCord

Assistant Commissioner for Coo~inator, Office of Research

Quality Assurance and Information Systems

New York State Education Department New York State Education Department

Cultural Education Center - Room 5A37 Cultural Education Center - Room 5B44

Albany, New York 12230 Albany, New York 12230

(5 18) 474-3896 (5 18) 474-6644

85
J

NORTH DAKOTA

The State Board of Higher Education seven-year plan directs each of the 11 campuses to have in

place, by 1996, a comprehensive program to assess student achievement of learning goals as defined in the campus mission statement. This is interpreted to minimally be the assessment process required by the regional accrediting association.

Additionally, the State Board has implemented a high school core curriculum admission require-

.:

ment for baccalaureate and graduate campuses and has contracted with ACT to study the impact

of this requirement. Feedback is provided to the high schools.

The University System is collaborating with other state agencies to develop a computerized tracking system for following college/university students into the workplace. The system will enhance

the assessment of college/university graduates’ educational experience for entry to and success

in the workplace.

Any further University System assessment activities will most probably incorporate the projects

described herein.

Contact: Gene A. Kemper

Vice Chancellor for Academic Affairs

North Dakota University System - -

State Capitol -600 E Boulevard Ave

Bismarck ND 58505-0230

(701) 328-2965

(701) 328-2961 (Fax)

e-mail: KEMPER@VMl .NODAK.EDU

86
?

OKLAHOMA

The Constitution of Oklahoma charges the Oklahoma State Regents for Higher Education with

responsibility for prescribing standards for admission, retention, and graduation applicable to

each institution in The Oklahoma State System of Higher Education. The Oklahoma State Regents also have the responsibility to provide leadership in the coordination of the orderly transfer

of students between and among institutions of the State System. Inherent in such responsibilities

is the prescribing of mechanisms to monitor and facilitate the assessment of students for purposes of instructional improvement and State System accountability.

Assessment in the state of Oklahoma has a multi-purpose objective. First and foremost, assessment is used to maximize student success by improving teaching and learning skills. Additionally, it serves an accountability purpose. Finally, student assessment assures the integrity of

college degrees and other educational activities/goals, to increase the retention and graduate rates

of college students, to enhance the quality of campus life in general, and to encourage high

school students to improve their academic preparation for college.

The Oklahoma State System for Higher Education uses a three part assessment system: entrylevel, mid-level, and exit-level. Additionally, collegiate-level assessment can be linked to primary/secondary level assessment for the purpose of improving student preparation.

.

The following is a excerpt from the Oklahoma State Regents’ policy statement on student assessment:

Each college and university shall assess individual student performance in achieving its programmatic objective. Specifically, each institution will develop criteria, subject to State Regents’ approval, for the evaluation of students at college entry to determine academic

preparation and course placement; mid-level assessment to determine basic skill competencies; exit assessment to evaluate the outcomes in the student’s major; and student perception

of program quality including satisfaction with support services, academic curriculum, and the

faculty. Such evaluation criteria must be tied to stated program outcomes and learner competencies.

An assessment fee has been approved which is designed to cover the cost of the assessment process. The fee is approximately one dollar per credit hour. Collection of assessment fees began in

Fall 1991 and generates approximately 3 million dollars per year, roughly 20 dollars per student

per year.

Entry-Level Assessment

The purpose of entry-level assessment is to assist institutional faculty and counselors in making

decisions that will give students the best possible chance of success in attaining their academic

goals. In Oklahoma, the ACT is used as the primary instrument for entry level assessment, and

the State Regents have set minimum performance standards required for admittance to the State

System of Higher Education.

The minimum ACT score required for entry-level admission varies by the type of institution. At

the two research universities in Oklahoma, students must score in the top 33 percent of ACT testtakers in the state. The ACT score equivalent to these percentages is based on the average of the

preceding three years’ ACT scores for graduating high school seniors. Currently, this equates to

87
J

a“n ACT score of21 or higher. At regional universities in Oklahoma, students must score in the

top 50 percent of all Oklahoma seniors tested. This currently equates to an ACT of 19. Two-year

institutions in the state have open admission. No minimum score is required, but the ACT or an

equivalent standardized test must be taken for the purpose of placement.

Each institution uses an established ACT score in the four subject areas of science reasoning,

mathematics, reading, and English as the “first cut” in determining individual student readiness

for college-level course work. The ACT subject scores provide a standard yardstick for measur4

ing student readiness across the State System. The subject score requirements are communicated

to college bound students, parents, and primary/secondary schools to make them aware of the

level of proficiency recommended in the basic academic skills. Students scoring below these levels are required to remediate in the discipline area or, consistent with institution’s approved assessment plan, undergo additional testing to determine their level of readiness for college-level

work.

Institutional entry-level assessment programs include an evaluation of past academic performance, educational readiness (mental, physical and emotional), educational goals, study skills, values, self-concept and motivation. Student assessment results are used in the placement and

advisement process to ensure that students enroll in courses appropriate for their skill levels.

Tracking systems are used to link assessment information and course work completion records.

These systems strengthen academic programs through evaluation and enhance student achievement.

Mid-Level Assessment

Mid-level assessments normally occurs after the student has completed forty-five semester hours

and prior to the completion of seventy semester hours for students in baccalaureate programs.

For associate degree programs, assessment may occur at mid-level or at the end of the degree

program. Examples of appropriate measures include academic standing, GPA, standardized and

institutionally developed instruments, portfolios, etc.

Generally, mid-level assessment tests competencies in the areas of reading, writing, mathematics, and critical thinking. Thus, the results of mid-level assessment improve the institution’s general education curriculum.

Program Outcomes Assessment

Program Outcomes Assessment, or major field of study assessment, is the third component of

the State Regents’ policy. Such assessments are designed to measure how well students meet institutionally stated academic program goals and objectives.

As with other levels of assessment, selection of the assessment instruments and of other parameters (such as target groups, when te_sting occurs, etc.) is the responsibility of the institution and is

subject to State Regents’ approval. Preference is given to nationally standardized instruments.

The following criteria are guidelines for the section of outcomes assessment methodologies:

88
a) Instrument(s) should reflect the curriculum for the major and measure skills and

abilities identified in the program goals and objectives;

b) Instrument(s) should assess higher level thinking skills in applying learned information; and

c) Instrument(s) should be demonstrated to be reliable and valid.

Nationally normed instruments required for graduate or professional study, or those that serve as

prerequisites to practice in the profession, may be included as appropriate assessment devices.

Examples are the GRE (Graduate Record Exam), NTE (National Teacher Exam), and various licensing examinations.

Assessment of Student Satisfaction

The perceptions of student and alumni are important in the evaluation and enhancement of academic programs and campus services. These perceptions provide an indication of the students’

subjective view of events and services which collectively constitute their undergraduate experience. Evaluations of student satisfaction are accomplished via surveys, interviews, etc. Resulting

data are to be used to provide feedback for the improvement of programs and services.

Examples of programs and activities to be included in this level of assessment are satisfaction

with student services, quality of food services, access to_finan~ial aid, resident hall facilities, day

care, parking, etc.

Primary/Secondary School Assessment/Accountability Feedback Systems

In addition to assessment systems at the college level, there are 102 primary/secondary school

districts (approximately 80,000 students) in the state which have voluntarily enrolled in the Oklahoma Educational Planning and Assessment System (OKEPAS). The purpose of OKEPAS is to

help primary/secondary educators to better prepare their students for advanced levels of education. OKEPAS is a service of the ACT corporation, and the fee is currently paid by the Oklahoma State Regents for Higher Education. The program consists of two series of tests that all

students at the participating districts are administered,

The first OKEPAS test, EXPLORE, is administered in the 8th grade. EXPLORE is used to assess a student’s readiness for secondary education and to start students thinking about educational decisions they will need to make in the years ahead. The second test is PLAN which is

administered in the 10th grade. The results of this test can be linked with both the EXPLORE

and ACT tests. Linkage to the EXPLORE allows student progress to be charted between the 8th

and 10th grade. The PLAN test also acts as a baseline from which student progress can be

charted between the 10th and 12th grade using the results of the ACT. Furthermore, the results

of all of these tests can be linked with students’ college performance data using the Oklahoma

State Regents’ Unitized Data System (UDS). UDS collects course level information on every student in the State System of Higher Education. This information is then relayed back to primary/secondary schools to keep them informed of their students’ progress once in college.

The Oklahoma Collegiate Success Projile provides feedback information to each high school site

in Oklahoma. These profiles are made possible through a cooperative effort between the Oklahoma State Regents for Higher Education and the ACT corporation. The report tells each high

school how many of their ACT-tested graduates enrolled in Oklahoma public colleges and the

type of institution they attended. The report tells how many of these students persisted beyond

89
>

their first semester in college. This information is displayed for students who went to college

with the required 11 units of core curriculum and those who matriculated without the core curriculum. The report also shows the rate at which students persist related to four ACT composite

score ranges as well as information on high school grade point average and college grade point

average.

The Oklahoma State Regents for Higher Education produce a series of reports for every high

school district in the state. Collectively, the three reports are titled The High School indicators

,. Reports. The first report provides information to high schools about the number and percentage

of seniors who have matriculated at an Oklahoma state college or university. This number is

used to calculate a high school to college going rate. The second in the series supplies each district and site with an ACT score based on the average performance of their students. Scores for

the county and state are also provided for comparative purposes. The third report provides the

first-semester average college GPA. Student performance is displayed in four GPA ranges, and

by county, district and site. This system helps schools accurately assess how prepared their students are for college course work.

Additional Elements of the Oklahoma Assessment/Accountability System

College Report Cards

The Oklahoma State Regents are exploring the possibility of producing an institutional report

card. The college report cards would be compiled for every institution in the State System for

Higher Education. It will likely include such things as attritiordretention rates, graduation rates

and job placement rates. These indicators could be calculated at both the institutional level and at

the program level.

The Athletes Graduation Rate ReL?or(

The Alhletes Graduation Rate Report is the result of a 1988 legislative mandate. It compares the

graduation rates of all first-time entering students to those who have athletic scholarships. This

report follows the methodologies set forth in the federal Student-Right-to-Know legislation.

Each fall, student cohorts are built which consist of all first-time, full-time, degree seeking students. These cohorts are then tracked annually to record student progress. Graduation rates can

be displayed by tier, institution, program, and race.

Placement Rates

The Oklahoma State Regents have been working with the Oklahoma State Employment Security

Commission to compute placement rates for graduates of the State System. This is accomplished

through the use of blind social security number transfer systems. The placement rates are calculated by program and include average wage information. Data sharing agreements are being

worked out with several of the surrounding states so that students will be logged even when they

receive employment outside the state. This project has been underway for one year now and will

help colleges and universities assess the relative success of their students once placed in the job

market. The data is still in a preliminary stage, and a report has not yet been produced.

90
Grade In flatlon Studv

The Grade Inj7ation Study was conducted in 1994. The purpose of the study was to compare

grade inflation in teacher education programs to other programs at the state and national level.

Grades were analyzed in relation to historical trends and students’ ACT scores. Considerations

are being made to continue this report in the future so that institutions can monitor their grading

practices in relation to other institutions in the state.

Conclusion

Oklahoma has made great strides toward improving its assessment and accountability feedback

systems over the course of the last ten years. This has been accomplished through the creation of

a three stage assessment system, the linking of college level assessment with assessment at the

primary and secondary level, and through the monitoring of such things as retention rates, graduation rates, job placement rates and grade inflation at the collegiate level.

Information is collected at all levels of education and is made available to those involved in the

educational process for the purpose of improving the educational system in the state of Oklahoma. However, due to the lack of comparable data at the national level, it is still difficult to

compare our state’s performance with that of the nation’s as a whole. It would be greatly beneficial to have NCES coordinate with the states to create common data elements relating to assessment and accountability measures. This information could then be collected by NCES and used

for state level comparisons.

Contact: Chancellor Hans Brisch

Oklahoma State Regents for Higher Education

500 Education Building, State Capitol Complex

Oklahoma City, Oklahoma 73105

Phone: (405) 524-9100

Fax: (405) 524-9235

91
PENNSYLVANIA

Origin/Interest/Authority of Postsecondary Student Assessment in the State

Pennsylvania has no mandate for assessment. Secretary of Education Eugene Hickok is promoting raising the level of achievement at all levels of education, and is exploring ways to provide

incentives for institutions of higher education to raise the standard for student performance.

.-< Primary Purpose of the Assessment Program

Not applicable

Brief Description of the Assessment Program

Not applicable

Means Used to Establish Validity

Not applicable

Common Data or Test Results

Not applicable

Reporting or Approval Requirements

Not applicable

Funding Available

Not applicable

Brief Overview of Findings

Not applicable

Special Assessment Needs

Not applicable

Comments

Staff and members of the State Board of Education, the Department of Education, the institutions and the legislature are attempting to become better informed on assessment.

.

92
‘J

SOUTH CAROLINA

Origins/Authority of Assessment in the State

Assessment was initiated through the Commission on Higher Education in a broad policy document called The Cutting Edge, which was largely adopted into law in 1988 (See State Code of

Laws Sections 59-104-650 and 59-1 04-660). Additional assessment requirements were added by

the legislature in 1992 (State Code of Laws Section 59-101-350). .+

Primary Purpose of the Assessment Program

The primary purpose of the original ( 1988) program was and continues to be the improvement of

educational quality, although accountability is clearly another purpose. Each institution is required to provide the Commission with an annual report on the results of its institutional effectiveness program and the Commission must prepare a report for the legislature. The additional

(1992) legislation requires a “report card” format and focuses more directly on accountability as

its primary purpose, although questions arising from the data often cause the institutions to examine issues of quality and improvement. The two required reports are combined into one document that is submitted to the legislature by January 15 of each year. Although legislative

committees often raise questions about findings that are reported, there is no direct relationship

between the results of the program and funding at this time. A proposal has been drafted that

would tie a small percentage of higher education funding directly to performance indicators.

Description of the Assessment Program

Updated Guidelines for Institutional Effectiveness provide directions for implementation and reporting under both the 1988 and the 1992 legislation.

Under the initial (1988) program, which was phased in over four years and is institutionally

based, the public colleges and universities use multiple methodologies to address assessment in

areas required by the Commission as follows: (a) general education; (b) majors or concentrations; (c) performance of professional program graduates on licensing and certification exams;

(d) program changes resulting from external evaluations; (e) academic advising; (f) entry-level

placement and developmental education; (g) success of matriculants in meeting admissions prerequisites; (h) achievement of students transferring from two to four-year institutions; (i) retention and attrition; (j) minority student and faculty access and equity; (k) academic performance

of student athletes; (1) student development; (m) library resources and services; (n) administrative and financial processes and performance; (o) facilities; (p) public service; and (q) research.

Institutions maintain a plan on campus and at the Commission offices that can be modified and

that defines each component (when necessary), indicates how it will be assessed, and provides a

reporting schedule (not all components are reported on annually). The assessment of institutional

effectiveness is a shared responsibility between administrators and faculty that must include

tracking of student progress through the curriculum and follow-up of graduates. A specific goal

is to “provide data. . . that can be used to initiate curriculum, programmatic, or policy changes

within the institution. ” Therefore, emphasis in the annual reports on this part of the program is

not only on findings, but on what actions the institutions will take or have taken to bring about

improvement as a result of the findings.

93
.-.

d

>.

South Carolina’s 1992 higher education report card legislation adds a requirement that the Commission report information and/or data in eleven specific areas (seven areas for two-year colleges) so as “to easily compare peer institutions in South Carolina and other SREB states with

the state’s public post-secondary institutions. ” Some of the eleven areas were already covered in

the initial program, but the new legislation added the requirement for comparative formats and

was generally more specific about data requirements.

The performance indicators that must be reported on in this portion of the annual report include

.-e (a) number and percentage of accredited programs and programs eligible for accreditation;* (b)

number and percentage of undergraduate and graduate students who complete their degree program;* (c) percent of lower division instructional courses taught by full-time faculty, part-time

faculty, and graduate assistants;* (d) percent and number of students enrolled in remedial

courses and the number of students exiting remedial courses ad successfully completing entrylevel curriculum courses; (e) percent of graduate and upper-division undergraduate students participating in sponsored research programs; (f) placement data on graduates;* (g) percent change

in enrollment rate of students from minority groups and the change in total number of minority

students enrolled over the past five years;* (h) percent of graduate students who receive undergraduate degrees at the institution, within the state, within the United States, and from other nations; (i) number of full-time students who have transferred from a two-year, postsecondary

institution and the number of full-time students who hav~ tra~ferred to a two-year, postsecondary institutions;** (j) student scores on professional examinations, with detailed information on

state and national means, passing scores over time, and the number of students taking each

exam; and (k) appropriate information relating to each institution’s role and mission.

There is an additional requirement that a survey on alumni satisfaction with the overall program,

the major, and general education as well as of societal participation be conducted biennially of

public institution graduates from three years prior.

Validity and Reliability

Inasmuch as measures of classroom teaching/learning are established by the institutions, and

most use multiple methodologies to assess this area, the establishment of validity and reliability

is specific to each method in general education or a major. The South Carolina Higher Education

Assessment Network, a consortium of 48 institutions and two state agencies, is available to provide technical assistance if needed.

Common Data or Test Results

Common data required under the 1992 legislation are described above. All institutions with students receiving athletic grants-in-aid also submit copies of their NCAA reports and limited supplemental information in response to item (k) of the original requirements. Common data are

also collected on the percentage of students who meet admissions prerequisites and on minority

enrollments, graduations, and faculty. Benchmarks are not pre-established, but results are reviewed in all areas and specific recommendations for improvements are made as appropriate for

the institutions’ missions.

* Same for two-year colleges

** Only two-year colleges report this item

94
Reporting Requirements

All 33 public institutions submit annual reports on institutional effectiveness to the Commission

by July 1 of each year. The Commission compiles data and institutional summaries and develops

a report that includes both tables and narrative information (300+ pages) for submission to the

legislature by January 15.

Notably, the narrative reporting process places responsibility on the institutions to describe how

.-e they interpret and use assessment data rather than just to report such data. This process is considered a key component towards translating simple reporting of information into actions for improvement.

Funding

Other than state support for the South Carolina Higher Education Assessment (SCHEA) Network, there is no special funding specifically designated for the institutional effectiveness program. The Network, which was initially funded through FIPSE, plays a major role in the success

of the South Carolina program, sponsoring an annual conference and workshops, publishing a

newsletter, maintaining a library of assessment publications and resources, and providing technical assistance.

Overview of Findings

.-

Findings on communication, problem solving, and thinking skills are generally distributed

through reports on the assessment of general education in the 33 individual institutional reports

in the Commission’s 1993, 1994, and 1995 Reports on Act 255 of 1992 and Summary Report on

Institutional E#ec[iveness. It is clear that most of South Carolina’s public colleges and universities consider communication and critical thinking skills to be among the goals of their general

education programs. However, they use a variety of instruments and methods to assess these areas, and it is extremely difficult to generalize with institutions that range from research universities to two-year technical colleges.

Special Assessment Needs

While we agree that the postsecondary assessment needs identified in National Education Goal

6, Objective 5 are critical, we believe that a much more broadly defined program such as that developed in South Carolina and a number of other states provides both the institutions and the

public with a better total picture of institutional strengths and weaknesses. We also believe that

emphasis should be placed on the collection of data that provide diagnostic information and can

be used for improvement rather than data that simply furnish a “rank” or “standing.”

Contact: Alan S. Krech

Associate Commissioner

SC Commission on Higher Education

1333 Main Street, Suite 200

Columbia, South Carolina 29201

Phone: (803) 737-2291

Fax: (803) 737-2297

E-Mail: AKRECH@CHE400.STATE. SC.US

95
J

1.

.+

2.

3.

4.

5.

?-

SOUTH DAKOTA

The South Dakota Board of Regents became interested in statewide assessment of academic

programs in the early 1980s following the development of a program at Northeast Missouri

State University. Program assessment began as a Board initiative with Board involvement

in the selection of instruments and the design of the program. Campus control of assessment

programs evolved over a number of years, with formal recognition of campus control occurring in 1987. In the 1990s, the Board’s requirement for assessment was merged with the

North Central institutional requirements for accreditation.

As part of an accountability initiative, the South Dakota Board of Regents is now interested

in guaranteeing that each of the graduates of its six universities has attained at least a minimum level of proficiency in a number of areas. A pilot program will be implemented in the

Spring of 1996 with a mandatory statewide rising junior examination to be implemented in

the spring of 1997. Students who are not able to demonstrate proficiencies will not be able

to continue to work toward a baccalaureate degree.

The focus of program assessment is to provide information to the institution so that adjustments can be made in curriculum and the delivery of the curriculum to improve programs.

The primary purpose of the student proficiencies program is to demonstrate that all baccalaureate graduates have attained at least a minimal level of proficiency in a number of areas.

The mechanisms for program assessment vary from campus to campus and from discipline

to discipline. Programs are assessed in a variety of ways including nationally -normed objective tests, faculty designed tests, capstone courses, portfolios, and student performances.

Often, incoming students are initially assessed and then re-assessed at the end of the sophomore year, but more commonly, in the senior year. While the methods of assessment vary,

the emphasis is on assessing student outcomes and answering the following questions: Are

students learning what the faculty think they are teaching? Have the students acquired the

knowledge, skills, and abilities identified in the programmatic assessment plans?

Student proficiencies will be assessed during the sophomore year. Current plans are that objective tests and an essay will be administered to all sophomores. The objective tests will

measure a variety of areas, including quantitative skills, problem-solving abilities, and scientific reasoning.

Since faculty are directly responsible for the implementation of program assessments, they

can directly link classroom teaching and desired learning outcomes with the program assessment method they select.

The goal of student proficiencies is to assure that all students demonstrate a minimum level

of proficiencies; therefore, student proficiency measures will be more directly tied to criteria references.

Program assessments are campus-controlled and vary widely across the state.

A single instrument will be used statewide to assess student proficiencies. Currently, the

ETS Academic Profile and the ACT Collegiate Assessment of Academic Program (CAAP)

instruments are being reviewed for adoption. Criterion referenced cut-off scores are expected to be adopted based on the recommendations of a statewide committee.

96
-.

#

6.

7.

8.

9.

The Board of Regents is expected to adopt student proficiency measurement requirements

in December. The result of student proficiency examinations will probably become public

documents that are used to report to the executive and legislative branches of government.

Each campus has been given permission to implement a student fee to fund program assessment.

A budget has not yet been approved for the student proficiencies initiative, but it is expected to include a combination of resources from the Board, from student fees, and from

institutional realignment of priorities.

The campus program assessment findings are diverse and difficult to summarize. A copy of

the reports provided by each campus to the Board of Regents in the Spring of 1995 is available.

The results of the student proficiencies initiative are not available since the program will enter pilot implementation in the Spring of 1996.

No special assessment needs over and above those identified by the national education

goals have been determined.

Contact: Dr. Michel Hillman

Director of Academic Affairs - -

South Dakota Board of Regents

207 East Capitol Avenue

Pierre, SD 57501-3159

Phone: 605-773-3455

Fax: 605-773-5320

Internet: mikeh@bor.state. sd.us

97
TENNESSEE

Assessment plays a vital role in Tennessee Higher Education. The roots of the current assessments go back to the mid 1970’s when the Tennessee Higher Education Commission began to

study the feasibility of basing some institutional funding on performance. That study led to the

Performance Funding Program, first implemented in 1979 which has put more than a quarter of a

billion dollars into institutional funding based on performance since that time. Total awards for

.-: the past several years have ranged from $25 million to $30 million each year. The standards under which finding is granted in this program are studied and altered as appropriate on a five-year

cycle. The initial standards and all succeeding modifications were accomplished with full cooperation and participation by campus and governing board personnel. Some parts of the standards

have been in place from the beginning. These include standardized tests of performance in general education, accreditation of major programs, student and alumni surveys, and other such

measures of educational quality. The current standards (1992-93 through 1996-97) and tables

outlining the history of the program in terms of points awarded and dollars granted since the beginning are available from the state.

With the passage of the state’s General Assembly in 1984 of the Comprehensive Education Reform Act, assessment entered a new era. This legislation, in addition to addressing all areas of

educational reform, specified certain “Legislative Benchmark&’ on which the Commission would

report each year. These benchmarks included standardized test scores, persistence to graduation,

professional licensing examination results, library holdings, funding for research and public service, and job placement in vocational programs. These reports were mandated for a period of five

years.

In 1989, the General Assembly adopted a document entitled Tennessee Challenge 2000 which established educational goals for the last decade of the century. Many of the reporting elements of

the Legislative Benchmarks were included in the goals which were based loosely on the report of

the Southern Regional Education Board’s publication, Challenge 2000. Reports of progress toward the goals established in this legislation are produced each year and distributed to the legislature and the educational community. Several legislative acts passed since 1989 have expanded

the content of this annual report.

The primary purpose of all assessment in Tennessee is the improvement of the educational process. Most of the assessment effort is focused on undergraduate instruction. A side benefit of the

assessment program is the increased legislative support which has resulted from such comprehensive reporting.

The assessment system in Tennessee is best envisioned as wheels-within-wheels, or perhaps

more aptly as gears-within-gears. Each campus has its own assessment program which, for the

most part, focuses on educational improvement at the classroom level. These results feed into departmental and divisional reports which in turn become college and university reports. All these

assessment results form a major part of each institution’s institutional effectiveness and planning

cycles. Appropriate parts of institutional assessments flow into governing board planning processes and a portion of those processes feed into the reporting system of the Tennessee Higher

Education Commission.

98
Contact: Dr. Donald Goss

Director of Academic Affairs

Tennessee Higher Education Commission

404 James Robertson Parkway, Suite 1900

Nashville, TN 37243-0830

Phone: 615-741-7565

Fax: 615-741-6230

.e

99
. .

Authority for Student Assessment

In 1985, the Texas Higher Education Coordinating Board (THECB) appointed the Committee on

Testing to determine how many Texas students entering public colleges were inadequately prepared for college-level work and to explore the feasibility of creating a basic skills test for these

.-$ entering students. The recommendations of the Committee on Testing were sent to the Texas

Legislature and during the 1987 session became law under Section 51.306 of the Texas Education Code (TEC). THECB Rules and Regulations, Chapter 5, Subchapter P “Testing and Remediation” implement the Texas Academic Skills Program (TASP).

Primary Purpose of Assessment Program

The primary purpose of the TASP is the diagnostic assessment of the basic skill preparedness of

entering students for college-level coursework in the areas of writing, reading, and mathematics

with the intent to improve undergraduate education and to improve retention to graduation. Test

results are sent to a student’s college or university. In the case of a failing score on a tested subject, the student is placed in mandatory remediation. Diagnostic scores for the tested skills and

subskills are included with the score report.

Description of Assessment Program .

TEC 51.306 requires that all students entering a public institution of higher education in the fall

of 1989 and thereafter take a reading, writing, and mathematics basic skills examination prior to

the accumulation of nine or more semester credit hours (SCHS) or the equivalent. Multiplechoice testing is used in all tested areas, though the writing test results are based mainly on a

written essay. The examination cannot be used for admission purposes. If skill deficiencies are

identified, the student is required to participate in continuous remediation until the student masters all sections on the examination. The student may not progress to upper division work beyond 60 SCHS nor graduate from any Texas public institution of higher education without first

passing all portions of the TASP Test. Institutions are required to offer advising programs for all

students and remedial programs for students with demonstrated skill deficiencies.

Validity and Reliability

Content committees are composed of college personnel selected from across the state to review

the test sections for content validity, Tested skills reflect preparedness to attempt college-level

coursework. Passing criteria are set by the THECB.

Common Data and Test Results

All public institutions of higher education report on their remedial programs through the CBM002 TASP Report to the THECB. Results are reported in the Annual Effectiveness of Remediation Report. Successfully completing remediation is determined by successfully passing all

sections of the TASP Test. Students requiring remediation are compared against students not requiring remediation for grade earned in first college-level mathematics and English classes,

grade point average, retention and highest award earned.

100
Reporting Requirements

TEC 51.306 requires all institutions to report to the THECB on the effectiveness of remedial and

advising programs. The THECB publishes statewide reports.

Funding

There is no direct state funding for the testing program, which is paid through the student test

fee. Two hundred thousand dollars is appropriated each biennium for student test fee waivers. <

Approximately $500,000 is appropriated to the Texas Higher Education Coordinating Board

each biennium to administer the program. The assessment program results in additional enrollments in remedial courses and non-course-based instruction.

Overview of Findings by Class Level

Not appropriate. \

Special Assessment Needs Identified

None,

Contact: Dr. David Gardner

Texas Higher Education Coordinating Board

PO BOX 12788 .*

Austin, TX 78711

Phone: 512-483-6150

FAX: 512-483-6147

E-mail: gardnerdd@thecb. texas.gov

101
Origins

>~-.

UTAH

A policy advocating institutional assessment to improve educational quality was approved by the

State Board of Regents in its 1986 Utah System of Education Master Plan. In 1992 the Utah Legislature put into law a statute mandating biennial reports on student assessment, faculty productivity, and program and facility measures. .-:

Purpose

The primary purpose is to determine how well the System is meeting its goals with regard to educating students. The assessment report was/is delivered to the Board of Regents and the legislative Education Interim Committee. The report is used to assess the quality of the education

enterprise and determine what other indicators would provide a fuller picture of accomplishments and impact of the System on the State. Individual institutions are expected to report to the

Regents their implementation of assessment procedures.

Description

The assessment is done biennially. A 1992 legislative mandate required assessment of students

which includes these indicators: admission scores, progress, mad outcome measures. Also requested are resource management indicators. These include faculty productivity, as assessed by

the number of hours spent teaching and in teaching-related activities, scholarly pursuits, research

funding, and on- and off-campus service. Measures of program effectiveness and facilities management are requested. These include program review and accreditation reports, library staffing

and acquisition data, and utilization of space information. The System added a section on the

Higher Education Technology Initiative. The progress of transfer students will be added for the

1997 biennial assessment and accountability report.

These data are gathered from State Data books and questionnaires completed by the individual

institutions.

Establishment of Validity

In the two previous statewide assessments, student progress was measured by retention and

graduation rates, employment information, and acceptance to graduate school. It is unlikely that

student scores will be included. Thus, there is no attempt to assess the teaching/learning transaction. However, as the System’s general education curriculum moves to a competency based system, methods of assessment will be developed and employed to determine if competencies are

learned and applied. The findings from the assessment are validated through triangulation; other

data sources support the findings. -

Test Results

Not applicable at this time.

Requirements

The assessment results are reported to the legislature and regents biennially. The Office of the

Commissioner oversees changes in development of the instrument, collection of data, and interpretation of analysis.

102
Funding

No additional funding has been appropriated. Staff time is used to refine the instrument, collect

and analyze the data. interpret the findings and write the report.

Overview

Learning outcomes on communication, problem solving, and thinking skills were not assessed.

Once competencies within the general education curriculum are adopted statewide, assessment

..+ instruments will be developed so that the competencies can be measured.

The assessment process is updated following each biennial report. A statewide taskforce reviews

the previous document, identifies areas that need to be included and/or altered, and revisits the

methodology and instrument used to gather assessment data. The taskforce includes representatives from academic affairs, student services, institutional research and the business community. All institutions in the System are represented.

Needs Identified

The Utah Assessment and Accountability report does not include actual student test scores in specific courses. However, it does include other student data such as the number of terms taken to

completion and the number of credits carried per term. It also includes student pass rates on national examinations, graduation rates, employment rates and zates of acceptance into graduate

programs. Besides these indicators, the current assessment report includes faculty productivity

measures, program accreditation and review data, and facilities utilization data. The impact of

the System’s Higher Education Technology Initiative is also included. The next biennial report

will include the initiative’s effect upon enhanced learning and student/faculty development.

Contact: Phyllis “Teddi” Safman

Assistant Commissioner for Academic Affairs

State Board of Regents

355 West North Temple -3 Triad Center #550

Salt Lake City, Utah 84180-1205

(801) 321-7101

(801) 321-71 OOFAX

psafman@cc.utahsbr. edu

103
..

Origins of Assessment in Vermont

Vermont has no statewide governing board and there have been no legislative mandates regarding assessment of student learning. Assessment efforts are largely directed by the higher education institutions and their own boards of trustees. Within Vermont, the Vermont State Colleges

and the University of Vermont function as distinct public corporations, each with their own .+

board of trustees.

Purpose of the Assessment Program

The primary purpose of the initial efforts to assess student learning within the Vermont State Colleges is to improve undergraduate education. Assessment information is used internally to identify points of concern and to identi~ curricular and student support areas which require further

inquiry.

The Assessment Program

The five colleges within the VSC currently assess entering students’ skills in three areas: writing,

reading, and mathematics. The results of these tests are used in student placement within the various curricular areas, not for admissions testing. In additim, the VSC tracks student retention and

program completion.

Assessment Instruments

The VSC currently does not use an instrument that links classroom teaching with assessment of

student learning on a systemwide basis.

Common Data Results

Vermont has no system to report common data or test results.

Reporting Requirements

There are no statewide reporting requirements. However, within the Vermont State Colleges, the

institutions undergo programmatic self-study and review by the VSC Board of Trustees on a

regular five-year cycle. Special program reviews may also occur outside this cycle.

Funding

There are no appropriations related specifically to assessment activities in Vermont.

Findings

None.

Special Assessment Needs

None at this time.

104
#

Contact: Dr. Joseph B. Moore

Director of Planning and Academic Affairs

Vermont State Colleges

P.O. Box 359

Waterbury, Vermont 05676-0359

phone: 802-241-2520

fax: 802-241-3369

e-mail: moorej@quark.vsc. edu

105
>..

H

..

VIRGINIA

Origins of Assessment Interest in the State

The legislature initiated an assessment policy in 1986 when it directed all public institutions in

the state “to establish assessment programs to measure student achievement” in Senate Joint

Resolution 83.

.-$ Primary Purpose

The primary purpose of the state’s assessment initiative is to stimulate curricular action and attention at the institutional level. In 1986, we hoped that the same mechanism could also serve to

make institutions accountable to the public.

Description of the Initiative

The legislative mandate directed the State Council of Higher Education in Virginia to spearhead

the effort by establishing assessment guidelines in cooperation with institutions in the state.

As a result, SCHEV published assessment guidelines in April 1987. The guidelines encouraged

institutions to use multiple indicators of assessment, such as:

●

●

●

Direct measures of student learning, including standardized tests and new faculty-developed

assessment measures (portfolios, senior projects, examinations, etc.).

Existing information, such as that pertaining to admissions, retention, and graduation rates;

community college transfer rates: licensing and certification examinations; job placement; and

alumni satisfaction.

Basic skills testing and evaluating the success of remediation

Institutions needed at least to assess students in the major and general education, survey alumni,

assess the success of remediation, and provide information on their graduates’ success to feeder

high schools and community colleges. The guidelines allowed the institutions to develop their assessment plans, but a gubernatorial directive established in May 1987 that institutions would be

eligible for incentive funding for the 1988-90 biennium only if they had an “adequate student assessment plan.” By June 30 of that year, less than two months later, all institutions had submitted

an assessment plan to the SCHEV. Although some required some revision, all were eventually

judged adequate.

Based on the institutional budgets, funds averaging $12 per full-time student were granted to the

institutions to implement assessment procedures. The institutions submitted progress reports in

1988, followed in 1989 by full repo& on the results of the first biennium of assessment. The

state now enters in 1995-96 its fourth full biennium of institutional assessment. Reports for this

biennium will be made orally.

Summary results of the institutional assessment efforts have been incorporated into the biennial

Virginia Plan for Higher Education. The plan reports that a wide scope of assessment activity

has been undertaken, with overall enthusiasm, generally strong commitment to the process, and

results that have been used, in many cases, to improve curricula.

106
f

The Means Used to Establish Validity

Each individual campus is responsible for establishing the reliability and validity of its measures.

Are Common Data or Test Results Collected Across the State?

There is no common collection of data, although efforts to track retention and success rates of

students transferring from two-year to four-year schools have resulted from assessment.

.e Is Reporting or Approval Required of Assessment Initiatives?

Originally, all assessment plans were approved by the SCHEV, working with a group of external

consultants, in the latter half of 1988. Annual progress reports do not require formal Council approval, but are reviewed and commented on by SCHEV staff. Biennial reporting of results are reviewed by Council staff and consultants from the institutions.

Funding

In 1988-90, the state appropriated $4.4 million ($2,962,100 from the general fund) for the biennium for institutional assessment. This money was, in the 1990-92 budget, incorporated into the

institutions’ base budgets, and institutions in many cases made significant additional contributions. But since the institutions have seen their general-fired appropriations decrease by over

$400 million since 1989, that money arguably has disappeared.

A Brief Overview of Findings

.

Given the program-by-program nature of the findings, they cannot be summarized. One general

finding of the programs is that general-education programs lacked coherence or point and, as a

consequence, could not be assessed. Virtually all have been revised to focus on learning goals as

a result.

Special Assessment Needs Identified in the State

While the legislative agenda initially turned on public accountability, the assessment agenda as

pursued by SCHEV is primarily focused on the improvement of undergraduate teaching and

learning. This has left the accountability needs unmet. The Council has attempted to fill the gap

through its Indicators of Institutional Mission project, which provide indirect measures of institutional input and output. It may also be necessary to have some general direct measure of student

learning. In cooperation with a number of other SREB states, the Council is looking into a test of

general intellectual skills that may be given to a sample of students across the state.

Contact: Margaret A. Miller

Associate Director for Academic Affairs

State Council of Higher Education

James Monroe Building

101 North Fourteenth Street

Richmond, Virginia 23219

(804) 225-2627

107
WASHINGTON

Origins of Postsecondary Student Assessment

In 1987, the Washington State Higher Education Coordinating Board (HECB) Master Plan challenged the public two- and four-year institutions of higher education to develop a multi-dimensional program of performance evaluation. The plan envisioned assessment as a link between

two separate but complementary goals: to improve the quality of undergraduate education and to

. . *

provide needed information about student outcomes to the HECB. Four initial means of assessing educational quality were identified:

●

●

●

●

Follow-up data on graduates’ work experience

Satisfaction surveys about students’ educational experiences

Surveys of employer satisfaction with employees’ college preparation

Pilot tests of nationally normed sophomore year tests of communication, comtmtation. and

critical thinking skills

A

After three nationally -normed tests (the College Measures Program, Collegiate Assessment of

Academic Proficiency, and Academic Profile) were piloted, the HECB concluded on the basis of

a task force report and evaluations by faculty members that the tests were not the best tools to assess the quality of undergraduate education. The Board modified the assessment plan to include

six common components:

●

●

●

●

●

●

Collection of entry-level baseline data

Intermediate assessment of quantitative and writing skills

End-of-program assessments

Program review

Alumni satisfaction surveys

Employer satisfaction surveys

The HECB initiative was reinforced by the Governor and the Legislature, which provided

$400,000 for assessment activities to each of the six four-year public institutions, and to the State

Board for Community College Education in the 1989-91 biennial budget. This funding was supplemented in 1990 by $60,000 for each of the 27 community colleges. (Five technical colleges

have subsequently been added to the system.)

Primary Purpose

The overriding goal of assessment has been to help faculty and institutions improve their programs, processes, and policies in light of student learning in degree programs, writing, quantitative skills, and critical thinking as well as other student outcomes (e.g., employment). The Board

encouraged each institution to develop its own approach to assessment, including adopting differ108
ent organizational structures, ways of implementing assessment, tests, and models for assessment

activities. Assessment has flourished due to its emphasis on faculty-driven assessment questions

and methods and its primary use as a means for self-improvement.

Assessment has been useful to institutions in allocating state resources, setting and measuring the

accomplishment of program or institutional objectives, and prioritizing among competing budget

uses. Both of these functions—assessment as improvement and aid to budget allocations—are expected to continue to be crucial in the future. .+

Brief Description of the Assessment Program

The six components listed previously serve as the common framework for the assessment program. Assessment efforts are designed by and tailored to the needs of each institution.

Means used to establish validity and reliability

Not applicable.

Common Data or Test Results

Not applicable.

Reporting or Approval Requirements of the Assessment Initiatives

An Assessment Coordinator appointed by each institution coordinates the assessment activities,

and acts as liaison to the HECB. Until 1995, institutions reported biannually to the HECB; now

only annual reports are required.

The Board stresses the use of assessment results in the preparation of institutional budget requests. (The HECB reviews, evaluates, and makes recommendations to the Legislature on operating and capital budget requests from four-year institutions and the community and technical

colleges.)

Funding Available

State appropriations for assessment during the 1993-95 biennium totaled $2,232,000 shared

equally among the six four-year public institutions ($372,000 each) and $3,725,000 shared

among the State Board for Community and Technical Colleges and the 32 community and technical colleges. The HECB receives no direct funding for assessment.

Brief Overview of Findings by Class Level

Examples of activities that have occurred on campuses, particularly in the areas of communication, problem-solving and thinking skills are available in the annual assessment report.

Special Assessment Needs identified in the state over and above those identified in the National Education Goals

The HECB is in the process of developing competency-based admissions standards to eventually

replace the current HECB minimum admissions standards. The new standards will be designed

to be consistent with the K-12 educational reform work of the Commission on Student Learning,

109
>..

to signal more precisely to students what it means to be prepared for college. and to allow students who have followed various pathways in high school to assess and demonstrate their readiness for the baccalaureate learning experience. A process for assessing the success of students

admitted under these new standards will need to be designed.

Contact: Kathe Taylor, Ph.D.

Senior Policy Associate

Phone: 360-753-7815

Fax: 360-753-7808

E-mail: kathet@hecb.wa.com

.

110
-..

WEST VIRGINIA

West Virginia colleges and universities have been formally involved in institutional and systemwide assessment activities since 1987 when the West Virginia Board of Regents named a Task

Force on Assessment. This group, comprised of representative members from the College and

University President’s Council, the state-wide Advisory Council of Faculty, the state Academic

Dean’s Council and Central Office personnel, developed a set of “Principles Which Govern and .+

Undergird Institutional and System Assessment Efforts” and “Institutional Guidelines” for campus assessment activities. During the ensuing academic year all higher education institutions prepared an inventory of entry-level, interim, and post-assessment instruments and activities while

the Assessment Task Force identified common instruments used by system schools and summarized the nature and extent of assessment data being collected on the campuses. The Task Force

gathered and distributed definitional information regarding assessment and detailed descriptions

of successful assessment programs and sponsored four institutional pilot assessment projects

most of which focused on entry level assessment and subsequent placement activities.

In 1989 the West Virginia Legislature realigned the governing structure for higher education in

the state. The Board of Regents was dissolved and separate governing boards were established

for the University System and the State College System of West Virginia. The Task Force on Assessment was re-named the West Virginia Higher Education Council on Assessment and continued to include membership from both systems. The Chairperson was retained but the

composition of the Council was changed to include a representative of each public college and

university. This Council’s initial charge was to develop a set of recommendations for each governing board establishing institutional responsibility for the development of assessment plans

and to develop a mechanism for the exchange of campus assessment data. Such recommendations were developed and adopted by both boards in the fall of 1989. The first recommendation

adopted by both boards stated that “Each public college and university is urged to develop a fiveyear comprehensive assessment program which is compatible with its mission and educational

objectives. Preliminary focus should be on learning outcomes. ” Soon campus assessment committees were established and the state-wide Council on Assessment initiated a series of workshops designed to facilitate campus efforts. Initial workshops were informational sessions

regarding CAAP, COMP, WORKKEYS, and BASE conducted by ACT and Riverside Publishing personnel. “Critical Components of the Campus Assessment Program,” (1991) a workshop

focusing on the development of assessment models appropriate for 2-year, 4-year, and graduate

institutions featured critiques by Dr. Trudy Banta, then Director of the Assessment program at

the University of Tennessee, Knoxville. Additional programs have included a Classroom Assessment Workshop (1993) featuring Dr. Thomas Angelo, Director of the Academic Development

Center at Boston College, and a fall 1994 workshop led by Dr. Rita Meyer, Vice-Chancellor for

Academic Affairs at the University of Minnesota, Crookston, regarding North Central Association guidelines for effective assessment programs.

The Council on Assessment, in cooperation with the state college Academic Deans Council, developed and approved a set of guidelines for the exchange of assessment data between campuses.

Thus, two-year colleges may receive information regarding the academic performance of their

graduates enrolled at public baccalaureate institutions and 4-year schools may also obtain information regarding the performance of their graduates in the state’s public graduate programs.

111
..

A primary goal of the state-wide Council is to facilitate campus assessment planning. Thus, early

on the Council prepared and distributed a brochure, “Assessment: The Faculty Role. ” The group

routinely provides information regarding current assessment research and conferences. The

Council Chairperson and representative members have participated in regional and national conference programs sponsored by AAHE, ETS, SAIR, and the North Central Association in order

to share information regarding proposed standards, strategies and concerns pertinent to assessment.

.- e

The Assessment Council has recently developed a Higher Education Graduate Survey Form designed to gather collective data from all students graduating from public colleges and universities in West Virginia regarding their perceptions of the quality of their undergraduate preparation

as well as information about their success in securing employment in-field following graduation.

The form has been recommended for implementation in the sp~ing of 1996.

Although the focus of both governing boards has been upon the development of autonomous

campus assessment programs, each has provided unique directives regarding expected institutional progress and areas of focus. The University System Board adopted a series of Assessment

Initiates in 1992 which emphasized its commitment to assessment and its general expectations

for institutional involvement and progress. University System institutions annually prepare a detailed report of their assessment activities as they relate to current assessment focuses determined

by the Chancellor and the Board. The University System has afso provided workshops, training

sessions, and follow-up activities regarding Total Quality Management.

The State College System has focused its attention on the assessment of general education. The

1990 Board of Director’s Plan for Quali~ and Accountability requires that “all freshmen entering state college system schools must demonstrate proficiency on standard tests in math, reading,

and English composition in order to be placed in college-level English and math courses. ” Subsequently, the Board of Directors of the State College System adopted ACT cut-offs for entering

freshmen in these subjects. Such standards remain in effect.

The West Virginia higher education governing boards and the state-wide Council on Assessment

have acted primarily to encourage and facilitate assessment activities rather than to issue directives regarding assessment planning. However, the West Virginia legislature has recently passed

two bills which include mandates for extensive higher education assessment activity. The First,

Senate Bill 412 (1992), the state report card legislation, directs the Higher Education Central Office to collect, analyze and report data regarding higher education in order to “make information

available to parents, students, faculty, staff, state policy-makers and the general public on the

quality and performance of public higher education.” The report, developed by the Central Office Research unit in concert with a highly representative state-wide report card committee, is a

comprehensive document which reveals both individual campus and systems’ progress and success. Among the specific indicators reported are the number of students served by tuition and fee

waivers, the average ACT and SAT scores of entering freshman, the number of degrees awarded,

the ratio of students to faculty, fiscal support for higher education, the number of applications

and acceptances to medical schools, and the number of medical school graduates in particular

spending areas. Of particular note is information regarding institutional retention rates.

Senate Bill 547, passed during the 1995 legislative session, outlines further requirements for institutional and systems’ assessment. The bill mandates that “the governing boards shall prepare

institutional report cards for institutions under their jurisdiction and system wide report cards . ..”

112
/

>-

and that “In assessing progress toward meeting goals and in developing trend information, the

governing boards shall review report card data in relation to previously adopted board goals, fiveyear plans, regional and national higher education trends and the resource allocation model.” In

addition to calling for data on academic performance, the legislation stresses the need for assessment of citizenship and human relations skills. The bill requires the development of institutional

and system strategic plans which focus on program review and fiscal accountability. Such plans

were recently provided to Central Office and governing board personnel for analysis and review.

.e

West Virginia governing boards have allocated approximately $15,000 annually for state-wide

assessment programs and materials. However, the primary responsibility for funding assessment

activity has been borne by the campuses. In 1993 the State College System Board provided a

one-time allocation of $5,000.00 for assessment activities for each campus under its purview.

Systems’ Chancellors have facilitated campus fiscal planning through their endorsement of a

group ACT-CAAP pilot project which provided reduced fees for test instruments to all participants.

West Virginia public colleges and universities have made substantial progress in the development of comprehensive assessment plans which are compatible with their missions and resources. Review of the campus programs suggest that they are characterized by strong

administrative support and faculty involvement, the designation of appropriate assessment instruments and activities to assess learning and personal grotvth afid commitment to utilization of assessment data to improve curricular offerings. Currently campus personnel are attempting to

integrate assessment activity into institutional planning processes while the state-wide focus is

on attempting to achieve a broader perspective regarding assessment by gathering and examining

collective data indicative of the preparation of the total state student population.

Contact: Dr. Suzanne T. Snyder, Chairperson

West Virginia Higher Education Council on Assessment

(304)-367-421 9

113
..- /

WISCONSIN

The University of Wisconsin System, with 15 institutions serving over 150,000 students,

adopted new accountability measures in 1993 on the recommendation of a special task force convened by Governor Tommy Thompson. The task force, composed of a diverse group from the

private sector, state government, the University of Wisconsin (1.JW) Board of Regents, faculty,

staff, and students, began its work by defining accountability as “being answerable for the effec-

.~ tive discharge of the mission of the organizations.”

Following a review of the system’s mission statements, the group identified three areas of particular concern to stakeholders: (1) delivering a high-quality undergraduate education, (2) meeting the needs of business and other organizations in Wisconsin and (3) being customer-oriented

and responsive to the concerns of customers, particularly students and parents.

Focusing on these priorities, the UW Board of Regents adopted a core set of performance indicators to demonstrate accountability in each area. In developing the indicators, regents were guided

by the task force’s recommendation that measures should be based on outcomes and should include a limited number both qualitative and quantitative indicators.

The 18 key indicators, which build on a more limited set of indicators already in place at the University of Wisconsin, are used to measure the performance of~he system in seven general areas.

These include effectiveness, efficiency, quality, access, diversity, stewardship of assets, and contribution to compelling needs of the states.

The following sample from the UW system’s 1994 report on “Accountability for Achievement”

describes key indicators, their corresponding performance goals, and progress as of September

1994:

. Faculty share of undergraduate instruction. The goal is to increase faculty involvement in

undergraduate instruction. Progress: Almost 70 percent of undergraduate instruction is delivered by faculty, down slightly from 70.4 percent in fall 1992; 23 percent is delivered by instructional academic staff, up from 22.3 percent in 1992; 5.6 percent is delivered by graduate

teaching and assistants, up slightly from 5.5 percent in 1992.

. Sophomore competency test. The goal is to exceed the national average in writing and mathematical skills. Progress: “UW students score above national average in writing and math

skills” as measured by the American College Testing-Collegiate Assessment of Academic Performance. This was the first year that UW students took the test; the test will be repeated in

the 1997-98 academic year.

. Minority student enrollment and graduation rates. The goal is to increase minority student

enrollment and graduation rates. Progress: Enrollments for African-American, Asian-American, Hispanic, and American Indian students increased by 4 percent between 1992 and 1993,

while minority graduation rates showed an overall 3 percent increase (comparing the eventual

graduation rates of the group that began in 1986 with the group that began in 1987).

114
.’

● Facilities maintenance. The goal is to reduce the maintenance backlog on institutional facilities. Progress: “Building commission has approved funds to reach 32 percent of maintenance

goal” as part of a ten-year plan to eliminate the $364 million maintenance backlog identified

in a building condition survey. The UW board has requested $126 million for 1995-97 to keep

the plan on schedule.

. Employer satisfaction with University of Wisconsin graduates. The goal is to improve the

.< career-readiness of graduates and the system’s responsiveness to Wisconsin businesses and

professions. Progress: “Employers rank UW graduates high” according to a survey of Wisconsin businesses and industry. The survey asked employers to rate graduates on basic skills (91

percent responded good or excellent) professional knowledge (83 percent responded good or

excellent), and critical thinking skills (91 percent responded good or excellent). Future surveys will provide a means for charting progress over time.

In addition to monitoring and publishing its performance measures, the UW system and each of

its institutions have developed processes (or refined existing ones) for enforcing accountability.

Among them are more explicit guidelines for post-tenure review, regular evaluations of all faculty and staff members, student evaluations of courses, and making summaries of the course

evaluations generally available to aid students in course selection.

Contact: Daniel P. Layzell

Director of Policy Analysis

University of Wisconsin System -

115
,.

The Origins

-..

PUERTO RICO

A major priority of the Council on Higher Education in Puerto Rico is the establishment of a systematic, statewide, formal assessment program though, at this time policies for such a program

have not been formulated. The economic feasibility of implementing such an assessment pro-

.-e gram is currently under investigation. The need for assessment programs, however, has been recognized by the Council on Higher Education as well as by both private and public institutions of

higher learning in Puerto Rico.

Primary Purposes

The primary purpose of assessment is the determination of thee ffectiveness of instructional programs in achieving their expressed outcomes. Knowledge of such outcomes aid in suggesting

curricular modifications, including the development of new educational programs which will

challenge students and increase the relevance of these instructional programs to the needs of society. Established assessment programs, for example, seek to reduce student dropout rates and enhance student achievement.

Brief Description of Assessment Programs .*

Assessment is conducted on different levels in institutions with assessment programs: the classroom level; the program level; the office or department level; and the institutional level. Survey

assessment is conducted in many of the institutions with assessment programs by using one or

more of the following approaches:

1. Portfolio Approach - Provides a continuous assessment of student achievement and faculty

effectiveness by collecting samples of student work for a specified period of time (e.g., a

semester, a year, etc.);

2. Reflective Diary - Students make notes on what has been studied during the week, reflect

on what has been studied, and discuss with the professor what was studied to determine if

the material presented has been understood.

3. Repetition - Students are asked to repeat in their own words the statements made by the

professor to determine if they have understood what was taught.

4. Three-Minute Paper - Students answer three questions about what was taught: what he understood, what he was unable to understand, and the aspects of the class he thinks need improvement.

The survey assessment approaches-described above are designed to assess basic, not complex,

student skills.

Measures Used to Establish Validity

Angelo and Cross’ Classroom Assessment Techniques: A Handbook for Faculty is used to establish validity and reliability in some institutions where assessment programs are in place and to

provide continuous and systematic evaluation of the efficacy of the assessment techniques used.

Student attainment of established objectives and the efficacy of instructional practices used to

achieve these objectives is assessed via the systematic collection of pertinent information. Such

116
J

goals are attained through the systematic adherence to a well-defined evaluation blueprint, the

collection of information on the practices essential to the achievement of these goals and a commitment to continuous and systematic examination of the outcomes of instruction. Data yielded

by such commitments form the basis for assessing instructional outcomes with educational outcomes.

Common Data of Test Results Collected Across the States

.< Statewide procedures for the systematic collection of data and program assessment have not

been established. Through the Puerto Rico State Postsecondary Review Entity regulation process, efforts were made to develop quantitative measures of effectiveness among postsecondary institutions. An Island-wide data collection system was designed and its implementation is still

under consideration.

Improvement of professor instructional skills is a common objective in institutions with assessment programs in place. Student evaluation of professors is a technique commonly used for this

purpose. Afier the student evaluation of the professor has been discussed with him, it is filed for

a limited time and eventually discarded.

Reporting of Approval Requirements of Assessment

Program review and approval of accreditation and licensure procedures applicable to public and

private institutions are used to assess programs in Puerto llic~. At the institutional level, individual professors may undertake assessment projects but such assessments are not mandatory. Assessment initiatives requiring financial support, however, must receive prior approval from the

program director.

Funding Available for the Past Three Years

Funding, though very limited, is available on a recurring basis for assessment programs in place

from unit implementing the assessment.

A Brief Review of Findings by Class

Communication skills, problem solving abilities, and critical thinking abilities are not specifically included in assessment programs in institutions with such programs. Student attitudes, satisfaction, and like variables are emphasized in assessment programs currently operating in Puerto

Rico.

Special Assessment Needs Identified

Critical thinking, problem solving, and communication are among the higher skills for which the

development of assessment instruments needed in assessment is under consideration in some of

the institutions engaged in student-evaluation.

Comments

The data reported here are based on a limited survey of the assessment practices employed in

some private and public institutions of higher education in Puerto Rico. Generalization of these

findings to a broader population is unwarranted. Nonetheless, the elements required for the initiation of a statewide assessment program appear to be in place in certain institutions.

117
All institutions of higher learning in Puerto Rico are accredited. Most of them are accredited by

the Middle States Association of Colleges and Schools (MSA), and are required to have assessment programs in place prior to receiving such accreditation. The Council on Higher Education

is contemplating the establishment of a master plan which will include the need for assessment

as an instrument for improving higher education in Puerto Rico.

Contact: Dr. Eusebio Diaz-Diaz

Planning Division

,_ *

P.O. Box 19900

San Juan, Puerto Rico 00910-1900

Phone (809) 724-7100 Ext. 2061

118
0-

APPENDIX C

TEXTS OF PREPARED CONFERENCE PAPERS

119
>..

WHAT POLITICAL LEADERS EXPECT FROM

POSTSECONDARY ASSESSMENT

Charles S. Lenth

Education Commission of the States

.~ Interest by this nation’s political leaders in postsecondary student assessment has been apparent

during the last ten years. Indeed, some observers have argued that much of the initial impetus as

well as the continuing motivation behind the assessment movement came from the broader interest in student “outcomes” and public accountability, as manifest in the calls of governors and actions of many state legislatures beginning in the mid- 1980s. At the same time, it must be

acknowledged that this political interest in assessment has been episodic, focused on quite different objectives, and often out-of-sync with the preferred practices and principles for assessment

within colleges and universities. This provides a dynamic tension within assessment between the

internal and external, between improvement and accountability, between formative and summative uses, and between those doing it and those who need to know.

My thesis in this brief paper is that just as political and other external expectations have shaped

assessment in the past decade, so these same forces will affect the purposes and directions of collegiate level assessment in the coming years-and appr~pria&ly so in my view. It is important,

therefore, to examine these external expectations for assessment to see where they come from,

how they differ, and what types of responses are most likely to be effective. It also seems useful

to explore these external expectations for assessment as a basis for speculating about what forces

are likely to shape collegiate-level assessment in the next few years. At least among state-level

political leaders, several major policy issues and societal concerns are likely to push their interest

and support for assessment in new and fairly clear directions in the coming years.

External Expectations in the 1980s

The wide dissemination of several reports by the Southern Regional Education Board, the Education Commission of the States, the National Governors Association, and other policy-oriented organizations in the early and mid 1980s put governors and legislators at the forefront of those

calling on colleges and universities to assess the learning outcomes of students. These, to be

sure, were not the only voices and motivators, but they were among the most prominent. Some

institutions such as Alvemo College in Wisconsin were already demonstrating approaches to

comprehensive, institution-wide assessment. Tennessee had established limited performance

funding that built in several types of state-wide student outcomes assessment under a rubric of

performance “criteria.” Georgia and Florida, followed by several other states, established or set

out develop some form of state-wide postsecondary student testing, either for admission/placement in college level courses or as a gateway, particularly for transfer students, to junior standing in baccalaureate programs.

For the most part, these precursors of what became a vigorous assessment movement were not

aimed at student assessment per se, but developed or adopted forms of assessment for other purposes. Early institution-based assessment at Alverno and a small number of other institutions

was part of an alternative model for a more integrated, student-centered and skill-based approach

to undergraduate education. In Tennessee, assessment was instituted not as a way to look at stu120
dent outcomes as much as a technique for allocating the small amount of non-formula-based

funding on the basis of some measure of institutional performance. In those states that established general testing programs, the purposes were grounded in concerns for student preparation

and readiness for college-level work. For political leaders, these were issues of basic educational

standards and fiscal accountability. In any case. these and other external motivators blended into

the assessment movement when it began to expand in the mid-1980s.

State Assessment Initiatives. Primarily in response to a perception that many public institu- ,<

tions, particularly large research universities, were disregarding teaching and learning in undergraduate education, several states undertook major initiatives to shift institutional resources and

attention in that direction. It is important to note that these state-level initiatives were motivated

by conditions both internal and external to higher education for which assessment was only one

of several policy approaches. New Jersey and Illinois illustrate this in different ways.

New Jersey initiated state-wide, college-level assessment in a more direct and comprehensive

manner than any other state. With the support of then Governor Tom Kean and substantial increases in funding from the legislature, Chancellor Ted Hollander and the Board of Higher Education initiated a major effort to enhance the stature and quality of New Jersey public colleges

and universities. In combination with multi-year competitive challenge grants and strengthening

institutional autonomy and governance, the College Outcomes Measurement Program (COMP)

provided for the development of multiple forms of student as.mssment and program evaluation,

part institution based and part state based. One component required the regular evaluation of the

effectiveness of all postsecondary remedial programs, with an eye toward improving these so

that all enrolled students could move quickly into college-level course work. The early results

from this evaluation and assessment identified not only the extend of remedial course work at

public colleges and universities, but the substantial variation in costs and effectiveness. The state

department also funded the development, by ETS and with substantial faculty involvement, of a

capstone college graduation assessment, the General Intellectual Skills (GIS) assessment. Based

on set of contextual performance assessments and administered to a sample of college seniors,

GIS was intended to as a tool for institutions to assess and improve their undergraduate education outcomes and for the state to evaluate the comparative performance of public colleges and

universities. Following initial development, the results from the first GIS assessment proved to

be unpalatable to institutional leaders and others. When changing economic conditions undercut

the financial support for this comprehensive institutional improvement initiative and state political leaders became less supportive, the entire set of state initiatives was terminated and set aside.

In Illinois, the Board of Higher Education adopted policies requiring institution-based student assessment as part of an broad initiative to improve undergraduate education. This initiative was an

attempt by the board and its staff leadership to “get ahead of the curve;” that is, to put into place

policies and a process of change to-help move public colleges and universities to address some

important external, public concerns before the pressure from outside became more vociferous.

The Illinois initiative was, however, considerably less ambitious and more decentralized than in

New Jersey. In particular, the undergraduate assessment component was to be institutionally

based and embedded in the established program review process. This approach tended to inhibit

institution-wide assessment of undergraduate outcomes, and produced very little comparative

performance data at the state level.

121
,.

The approaches and results in both New Jersey and Illinois were much more complex-and ambiguous—than these brief overviews convey. My purpose is only to draw some tentative observations about the dynamic between the external, political forces and the directions of collegiate

assessment at the state level. First, whether quite forcefully as in New Jersey or more subtly as in

Illinois and a number of other states, the purposes of state-level assessment initiatives were defined primarily by or in anticipation of the needs in the external environment of higher education. In retrospect, the differences across states were mainly those of strategy and stage of

development, not those of longer-term objectives. State based assessment itself has been a formative rather than a sumrnative process, with objectives that emerge and are clarified as a result of

the accumulation of experience and the evolution of new techniques. This developmental metaphor it seems to me is important to thinking about the purposes of assessment in the future, and

its relations to externally defined needs and expectations.

My second, related observation is that one must also view the past and future of assessment in

terms of its practical effectiveness, if you will its developmental strategy. Here too, as with thinking about its objectives, the external context and political environment are very important-perhaps even more so than realized at the time. The comprehensive and ambitious assessment

initiative in New Jersey was terminated when state financial support was no longer available,

and when political support was undermined by institutional opposition, among other factors. The

more incremental Illinois approach lived on, not because it was any more successful in establishing assessment per se, but because it led to several other strategic steps. Arguably, at least,

the more decentralized and incremental assessment policies adopted in Illinois provided the time

necessary for the state agency to establish its roles in certain core assessment areas and for the

state political leadership to become comfortable with the necessity and directions for change in

higher education. Specifically, the Illinois Board of Higher Education had sufficient time to involve the institutions in the development of state-level student assessment data systems (both a

high school feedback report and a postsecondary student unit record system), and to embed assessment in a much more comprehensive and consequential initiative in the early 1990s, the

Board’s Priorities, Quality and Productivity initiative.

Patterns and Discontinuities. Looking back at the dynamic between assessment and the external environment during the 1980s, it is more helpfi.d to look for patterns than discontinuities. The

episodic nature of political leaders’ interest in and support for assessment is clear; differences

across states and periods are even more obvious. More interesting, however, is to look for patterns in the objectives and in the types of support political leaders seem willing to provide. Without going into detail, it seems to me state political leaders have demonstrated a pattern of support

not so much for assessment in particular, but for using assessment and other techniques as ways

to get colleges and universities to examine more closely how well they are meeting student and

societal expectations. This interest has been relatively constant for at least the last decade, although as I will argue below it evolves and becomes more focused to reflect what those external

needs are.

Political leaders’ strategic and developmental interest in assessment is even more difficult to

grasp and grapple with. But looking back in this area as well I think one can discern some patterns along with the obvious discontinuities. Political leaders often do not understand the complexities and intricacies of developing new ways to assess the effectiveness and outcomes of

college and other postsecondary education. While there are some obvious examples of misunderstanding and impatience, it seems to me there is a pattern of fairly consistent interest and support

122
>.,

for the exploration and development of new assessment methods. Conflict has resulted, it seems

to me, not as much from the impatience of political leaders as from the lack of responsiveness

from the higher education community to this developmental agenda. Those political leaders who

have advocated and supported assessment have fairly consistently recognized that substantial

technical and infrastructure development will be needed. Their response, by and large, has been,

let us get on with these tasks. This is a position in favor of assessment, not in opposition, although it does throw the challenge of further development back on higher education leaders and

.= the assessment movement.

Looking to the Future—What Will Political Leaders Expect?

To the extent that political leaders, particularly at the state level, shape the purposes and support

the development of new forms of postsecondary student assessment, how will they attempt to influence the directions of assessment in the future? My answer is impressionistic at best, even

more impressionistic and speculative than interpretations of these influences in the past decade.

To the best of my knowledge there are no wide scale survey data or other hard evidence on the

attitudes and expectations of political leaders toward assessment. Nor it is a topic that is foremost

in their minds or prominent in their speeches. My tentative observations are based on some unsystematic interviewing and focus group conversations with various state political leaders over

the past year or two, more extensive and personal contact with a small number of governors and

legislators, and some exposure to their public speeches,-pres{ accounts and other sources. These

limitations notwithstanding, I will argue that state political leaders are now and will increasing

become interested in how postsecondary assessment can and will link to three larger issues.

Linkages and Responsiveness to K-12 Standards. The most compelling of these is how to link

collegiate level assessment with the related and already more powerfid changes affecting K-12

education. These potential linkages are visible to political leaders in several ways. First, although

the National Education Goals adopted by the collective action of the then-President and state

governors in 1989 are no longer given much attention, the standards-based-education movement

and other reform efforts both leading to and pushed forward by this unusual collective political

action are very much alive. The notion of setting goals and monitoring progress in education has

yielded to a more sophisticated agenda-setting and implementation approach involving:

1. a process for establishing standards that reflect leaning and performance expectations for

students at various levels;

2. the development of new, more authentic ways to assess and give credit for achieving these

outcomes; and

3. the first exploratory steps toward building these standards and new assessments into the education system through restructuring the curriculum, teacher education and professional development, school performance indicators and other means.

The realization, even piecemeal, of this systemic reform effort in K- 12 education will require

substantial adjustment and specific types of support from higher education. These adaptations

are already underway in Oregon, Florida, Wisconsin and other states in initiatives to change college admission policies to be more parallel to standards-based K-12 education, to reform teacher

education and professional development, and to redesign college curricula and teaching practices

to be more congruous with current student expectations. Perhaps even more important, in the

minds of political, business and community leaders who have been involved in these education

123
,.

>..

reforms, the logic of this reform agenda appears to apply equally well to higher education. If explicit student learning and performance standards work well in the secondary environment, why

don’t these apply to postsecondary education equally well? When we have new means to assess

performance against these standards, will assessment at the collegiate level take on more of these

authentic, performance-based forms? When we have more explicit standards and advancement

through demonstrated performance up through high school and to the point of college admission,

can we apply this same logic to determining student progress through college degree programs

----

as well?

These are among the questions and insights that are beginning to come to the minds of political

leaders and others. A very prominent example in the “P-16 Initiative” of Governor Zen Miller in

Georgia, in which the state higher education system leadership is a primary partner. Less visible

but more numerous are the regional K-16 initiatives at several sites around the country, and the

substantial involvement of higher education faculty in the development of K- 12 standards.

Clearly, there are many implications and roles for postsecondary assessment in addressing the initial linkage issues and, eventually, the more fundamental restructuring challenges of standardsbased reforms. Comprehensive standards frameworks are still a good way off for K-12

education, and surely farther yet for postsecondary education. I believe, however, that the pressures and rationale for collegiate assessment to be explicitly linked to performance standards and

assessments at other levels will continue to grow. Before higher education sees systemic standards reform, it is very likely to see innovative institutional leaders take up the call of political

leaders to design and provide models of standards based postsecondary education, with new

types of assessment being key to this.

Value-For-Money Assessment. A second expectation that political leaders appear to hold for assessment is that it should address questions of cost effectiveness—what student outcomes are

achieved at how much cost. This represents a blending of the demand for fiscal accountability

for the expenditure of public finds with the value-added framework for pre- and post-assessment

of the learning that takes place during college. As the concept of value-added assessment becomes better know, this connection is made by political leaders without realizing that techniques

are not yet available to measure either the educational gains or the precise costs involved, let

along to combine these into a value-added-per-dollar-expended assessment. Despite this, the expectation seems to be growing that such an assessment framework would be desirable in the future. Both the sharply rising costs of college education and the constraints on public tax support

and subsidies lead in this direction.

Given these expectation and the likelihood of an increasingly constrained fiscal environment,

state political leaders will continue to push assessment in this direction. One way to attempt to

address this expectation is through developing some indirect linkages between financial data and

educational outcomes. Private colleges and universities already do this in informal ways, arguing

that while college “X” costs more, what students get for their money in four years is worth the

cost. Public institutions tend to argue only the low cost side, without much information about

what student get for their investment, how long it will take, and what long-run returns are Iikel y.

Despite the technical difficulties and lack of comparative statistics, I believe that political leaders

and others are likely to push college assessment in this direction of cost -effectiveness.

Assessing Learning Outside the Classroom. A third and growing influence on assessment will

124
J

involve assessing learning that occurs outside as well as inside the classroom and campus. To

many governors and legislators, this seems quite obvious and necessary. Witness the agreement

by the western governors in November 1995 to explore the potential for a multi-state “virtual university” that could award credit for learning no matter where it occurs. Other governors are committed supporters of workplace and applied learning options, beginning with high school

students but extending logically to postsecondary learning as well. In either case, the scope and

functions of assessment need to change substantially to include forms and locations for learning

.e outside of traditional classroom environments.

A colleague put the challenge this way, Suppose that we are consultants to the founding president of a new university. He or she wants our advice on the needs of this new university, how it

should be set up and, specifically, what kind of student assessment system to establish for admission and student advancement. The only thing the founding president currently knows is the location of the new university; it will be located right alongside the information superhighway. In

this location, much of the available knowledge base and learning resources will be out on the information highway, rather than in the traditional classroom setting. Students will come with specific education and training needs to augment the knowledge and learning gained elsewhere.

Faculty will be oriented to the world-wide resources, not to those held on campus. Libraries will

be connecting points, not stable collections. And assessment of student learning maybe more important and challenging that ever before, since unless it takes into account the knowledge and

learning from elsewhere, the new university’s contributions to student learning will be small and

marginal indeed.

Individuals from outside higher education, and specifically many political and business leaders,

seem to grasp immediately the implications of this new university. This may be the world in

which they already fimction, and where they see the future. To accommodate this world, student

assessment and the granting of credentials and degrees will need to take into account the large

and growing proportion of learning that takes place outside the conventional boundaries of classroom and campuses. Assessment will no longer be classroom and campus bound, but will need

to be linked across physical locations and with the learning and resource networks of the world.

How fast and how far assessment will be able to move in these directions cannot be certain, but

the reasons and pressures to move are already being articulated by political leaders and will continue to be felt both within and from outside the traditional academic framework.

Concluding Thoughts

In this challenging and rapidly changing context, assessment of postsecondary student learning

needs to take on broader and more central roles for colleges and universities. If not, like those

very institutions, what we now know as collegiate assessment will become increasingly separate

and marginalized in relation to the-spread of knowledge and learning in society. Looking back at

the last decade of development, the interest and pressure to integrate assessment into the classroom and throughout the campus has been much stronger than the external determination to establish new forms of external assessment. I believe that this is in the process of changing. The

long-standing tension between the internal and the external, between formative and summative

uses, and between those who do assessment and those who need to know the results is about to

enter a new era. The healthy and creative tensions will not go away, but the external forces will

become more compelling than ever.

Given this environment, higher education needs to prepare for and participate in the develop of

125
,.

more independent, comprehensive and authentic types of learning assessment that will augment,

but not necessarily replace, the assessment that now occurs in classrooms, across campuses and

programs, and to a lesser extend across states. Some political leaders are already reflecting these

future needs as they push higher education to pay closer attention to questions of standards and

measurement of student learning at all levels, when they asked now unanswerable questions

about value-for-money, and when they plan virtual universities around or in place of traditional

institutions. Assessment, long disregarded and frequently marginalized, may turn into one of the A- core functions of postsecondary education in the future. This is the direction in which many political leaders and other external influences are and will continue to push postsecondary education.

.

126
/-

/

PERSPECTIVE FROM THE NATIONAL EDUCATION GOALS PANEL

Ken Nelson, Executive Director

National Education Goals Panel

Thank you for your attendance and willingness to tackle tough and critical issues. Special thanks

.e to Sal Corrallo for his untiring efforts to enhance the postsecondary experiences of all students.

Einstein said, “We can not begin to solve the complex problems we face today with the same

level of thinking we had when we created them.” A new level of thinking is required of this

workshop, of you and your state to solve the challenges of postsecondary assessment. It was a

new level of thinking that created the National Education Goals.

Gov. Romer, a member of the Goals Panel but speaking as a parent, has often asked, “ Where do

I go to find the appropriate, quality postsecondary education for my children? Where can I find

the right information about what value respective postsecondary institutions will add to my

child’s learning?” How would you answer the Governor’s question?

Role of the Goals Panel

The role of the Goals panel is to monitor and report on progress the nation and states are making

toward achieving the national education goals. The Panel would like your assistance to achieve

Goal 6, Objective 5, and that’s what this workshop is all about; to determine how we can best assess, monitor and report on postsecondary academic achievement.

Goal 6 reads “By the year 2000, every adult American will be literate and will possess the knowledge and skills necessary to compete in a global economy and exercise the rights and responsibilities of citizenship.”

Objective 5 of Goal 6 reads: “The proportion of college graduates who demonstrate an advanced

ability to think critically, communicate effectively and solve problems will increase substantially.”

History of the Panel’s Recommendations

I. As you perhaps know in July 1990 the Goals Panel was created to develop a mechanism to

monitor and report on the nation and states’ progress toward meeting the goals. Traditionally the Panel has selected “some of the best minds” in the country to form a Resource

Group which makes its recommendations to the Panel.

A Resource Group on Adult Literacy and Lifelong Learning was formed which included

Mark Musick as the convener and Peter Ewell and Paul Barton as members. Its recommendations on Goal 6 indicators were presented in a discussion document to the public in regional forums. The Discussion Document was then finalized on March 25, 1991 and

entitled “Measuring Progress toward the National Education Goals: Potential Indicators and

Measurement Strategies.”

These germinal recommendations were made: “Assessing the Knowledge and Skills of

Graduating Seniors. The Resources Group notes that if the National Education Goals Panel

wishes to assess directly the ability of college graduates to think critically, communicate effectively and solve problems, a new kind of assessment will have to be created. (The Re127
> . .

,.

source Group considers both the NALS and the Graduate Record Examinations to be inappropriate for this purpose). Suggesting that developing such an assessment (which could be

modeled on the NAEP) would be both complex and controversial, the Resource Group estimates investment costs of “several scores of million of dollars and 5 years or more of development work for the system to become operational. ”

“A NAEP-like examination of graduating college seniors would be necessary if the Panel is

serious about measuring the advanced thinking, communication and problem-solving skills

of the Nation’s college graduate. ” (In November, 1991 The National Center of Education

Statistics convened a study design workshop to examine means of assessing college student

learning in support of the sixth National Education Goal.)

II. In February 1992 a new Task Force was formed to focus more precisely on “assessing the

national goal relating to postsecondary education. ” This time Mark Musick and Charles

Lenth were among the advisors and David Longanecker among the members. Part of its

charge was to investigate and report on “the feasibility and desirability of a sample-based

collegiate assessment which would provide regular national and state representative indicators of college graduates’ ability to think critically, communicate effectively, and solve

problems.”

In a report published July 31, 1992 the Task Forcemadesix recommendations to the Panel:

1) encourage the development of a sample-based national system of standards and assessment for postsecondary education; 2) suggest that content and performance standards be developed for general cognitive skills, higher order thinking skills, and occupational specific

skills where appropriate; 3)insist that in order to maximize their usefulness, assessment efforts be better coordinated through a formal structure; 4) urge the Secretaries of Education

and Labor to approve funding for assessment and skills certification activities only if the activity is coordinated and recorded in the inventory of assessment activities; 5) recommend

the creation of a separate coordinating council for postsecondary standards and assessment

that parallels those recommended by the National Council on Education Standards and Testing for elementary-secondary education and recommend financial support from the Congress to support this activity; 6) establish as an objective the development of a constellation

of indicators of postsecondary performance which includes basic skill levels, occupational

skill levels and higher order skills.

In summary, the Task Force concluded that a postsecondary assessment system would be

both desirable and feasible, and recommended that content and performance standards be

developed for general cognitive skills, higher order thinking skills, and occupational specific skills where appropriate. (In November, 1992 NCES convened a second study design

workshop to discuss which skills should be assessed in a collegiate assessment, how to select standards, and other measurement issues.)

III. In April and May of 1993 the Goals Panel conducted a series of regional hearings in order

to obtain input and public comment from higher education representatives, faculty, students, etc. In July 1993 the Goals Panel made its recommendation on Postsecondary Assessment: “The National Education Goals Panel believes that it is both feasible and desirable to

develop a national sample-based postsecondary assessment system that will provide regular

national and comparable state indicators of college graduates’ ability to think critically,

communicate effectively and solve problems and that includes assessments of occupation128
.-e

specific skills for students in occupationally-specific programs. ” Therefore, these general areas of concern were identified and incorporated in a drafl resolution.

1.

2.

3.

4,

5.

A national sample-based system of assessment for postsecondary education should be

developed. Such a system should encompass the differing institutional missions within

the postsecondary sector. This national assessment system will not provide information

about the performance of any individual postsecondary institution or student.

To promote the highest levels of performance throughout postsecondary education, the

assessment system should be driven by rigorous content and performance standards.

In assessing students’ abilities to think critically, communicate effectively and solve

problems, the system should be designed to reflect students’ differing fields of study

and occupational areas.

A broad-base, consensus building process should be used to set appropriate standards

and achievement levels for this postsecondary assessment system and to review and

evaluate assessment approaches. The consensus building process should involve faculty and administrators representing a variety of institutions (varying in mission, geographic location, etc.) as well as employers, policymakers, institutional researchers,

assessment experts, and higher education coordinating boards.

A variety of regularly reported postsecondary performance indicators should be developed from this assessment system. They should be chosen for their ability to provide

usefi,d information to different audiences including policymakers, system participants

(e.g., administrators, faculty and students) and the general public.”

IV. The Goals Panel was hopeful that NCES would take it the next step. As our invitation to

this conference indicated in its background information, “In 1993 NCES prepared a Request

for Proposal that focused upon skill identification for the college graduate. The purpose was

to identify and obtain a consensus of the skills and levels of competency needed by college

graduates beyond college. The results were to be used as a basis for the development of a

national survey of college student learning. A primary use of the information was to be used

on the status of efforts to meet the national education goals. The RFP was not fimded due to

budget constraints and it is unlikely that NCES will soon have sufficient resources to conduct a comprehensive national study, as originally envisioned. As a result, alternative strategies were developed.” And that brings us to this conference.

V. I want to illustrate what indicators for Goal 6 we report and how we do our reporting. This

might be helpfil as we think about state and local reports.

1995 DATA VOLUMES: The National Data Volume, page 118-132, Goal 6 indicators,

with Exhibits 56 and 57 provide direct measures of the Adult Literacy; and Exhibits 58 and

59 on Workforce Skills. Exhibit 60-62 provide direct measures of the Objective, Opportunity to Acquire Knowledge and Skills; and Exhibit 62 updates us on Postsecondary Enrollment and Completion. On pages 183-186 “The Technical Notes and Sources,” provide

background information.

The State Data Volume provides for each state the direct measures of the Goal on Adult Literacy and Citizenship as well as a direct measure of the Objective on Postsecondary Enrollment. The supportive Technical Notes and Sources are found on pages 256-258.

129
.-

1995 CORE REPORT, page 23, core indicators 16, 17, 18 provide information on Adult Literacy, Participation in Adult Education and Higher Education; graphically presented on

pages 47-49. In addition, you might find this interesting. On page 57 we present our National Data Collection Schedule and on page 60 our State Data Collection Schedule documenting a dramatic need for more data at the state and national level.

1995 EXECUTIVE SUMMARY, page 8 provides information on the progress we are mak-

.>- ing on three indicators for Goal Six: Adult Literacy, Participation in Adult Education, and

Participation in Higher Education.

IN CONCLUSION. During its entire history the Goals Panel has demonstrated its deep and

abiding commitment to provide the best data possible on the nation and states’ progress toward realizing Goal achievement. But it is apparent that not enough data is available to effectively report on the progress of the nation and states’ postsecondary education. As

Einstein said, “We cannot begin to solve the complex problems we face today with the

same level of thinking we had when we created them. ” Hopefully, this conference on postsecondary assessment will provide the new level of thinking that will enable the Goals

Panel to report “the proportion of college graduates who demonstrate an advanced ability to

think critically, communicate effectively, and solve problems (has increased) substantially.”

130
THE STATE PERSPECTIVE

Mark D. Musick, President

Southern Regional Education Board

I have been asked to provide a perspective on “state assessment in the current political climate.”

.e First, let’s acknowledge that there are at least as many different state perspectives as there are

states but I do have a perspective or a diagnosis, if you will, about assessment in the current political climate. My diagnosis regarding collegiate assessment is that the current climate is producing schizophrenia or perhaps it is only double vision. My conclusion, however, is that whichever

of the schizophrenic split personalities you take, or whichever of the double visions you view,

the conclusion is basically the same—you will see more emphasis, not less, on assessment at the

college level in the years ahead.

As a part of the political split personality about higher education, I would argue that there is

strong public support for higher education and a very strong belief in the value of higher education. I have yet to see a poll that doesn’t show that the American public overwhelmingly believes

that higher education is important, that parents want their children to continue their education beyond high school, and that Americans believe that education beyond high school is important to

their fiture, the future of their children, and the future OY the ;ountry. Colleges and universities

remain near the top of the list of institutions in which people have confidence. (Since we are in

Washington. it would be appropriate, I suppose, to mention that these are the same polls that

show Congress at the bottom of the list of institutions in which Americans have confidence.)

Having said that, let’s acknowledge that the level of criticism of higher education among the public, if not at a high point, is certainly high enough to get the attention of nearly anyone who has

anything to do with higher education. In this age of information overload, the public hears

weekly about something stupid or at best an action showing questionable judgment that has happened on a college campus. And we are not talking about student pranks. I don’t need to recite a

litany of indiscretions, poor judgments, or in some cases downright fraud, that have occurred on

our campuses. I don’t know whether there is more or less of these than there used to be, but we

certainly know more about them than we used to.

Where the political split personality about higher education comes into play is that there has

never been more support, certainly not in my lifetime, for assessment at the college level, and

that support is from the critics and supporters of higher education. This is not simply a Democratic/Republican thing. I think an impartial observer would say that Republicans have generally

been more critical of some of the ways higher education does business and have argued more

pointedly for greater emphasis on productivity and accountability. We have significantly more

college-level assessment in the South today than we had ten years ago. That all occurred under

state houses controlled by Democrats and, for the most part, under Democratic governors. Now,

I know there is a difference in a Southern Democrat and a national Democrat, but I would argue

that generally where we have college-level assessment programs, these have had the support of

the political center.

131
/.

The most fundamental reason that I believe that the current state political climate will produce a

growing emphasis on postsecondary assessment is that both critics and supporters of higher education are focusing on accountability. Without trying to categorize critics and supporters, think

with me quickly about some of the things that state leaders are saying.

Recently in Alabama, the Alabama Commission on Higher Education and the state Board of Education sponsored an Education Summit. The Governor raised a lot of questions about higher education-a lot of open-ended questions. It seemed like he had reached some answers in his own

,--- mind, chiefly that higher education was getting more money than he thought might be justified

and that some kind of collegiate assessment was needed.

A task force in Kentucky was established by the Speaker of the House, who, unquestionably, is a

friend of higher education. The Speaker established the task force in large measure because he

felt that higher education was inadequately funded. But one of the initial questions he posed, and

I quote, is “Kentucky needs a better method for determining the success of our colleges and universities in turning out qualified, knowledgeable graduates. How can we devise a better model?”

The Governor of Arkansas ran on a platform that included reducing remedial education in college and of having a rising-junior assessment.

Colorado Governor Roy Romer, a member of the National Education Goals Panel, has suggested

that higher education needs a “Consumers Report” or a “3. D. Powers’ survey.” (For those here

not into automobiles, J. D. Powers rates automobiles). Needless to say, Governor Romer’s suggestions relate to assessment.

If you read the Chronicle of Higher Education, you may have read about the chairman of a special education study committee in Pennsylvania who was questioning higher education on almost

all fronts, including its quality. According to the Chronicle article you either think that legislator

is asking needed questions about institutional priorities and faculty productivity or you believe

that he is diverting attention from the higher education issues that will help move the state forward.

The Wingspread Group on higher education said, “The simple fact is that some faculties and institutions certify for graduation too many students who cannot read and write very well and too

many whose skills are inadequate in the face of the demand of contemporary life. ” A pretty obvious implication is that more or better assessments are needed to see that students do read and

write well enough and do have adequate skills.

Public policy institutes are being formed in many states. Generally these institutes have a conservative agenda and are often focused on education. Most of the education emphasis of these institutes has been at the elementary/secondary level. Some have called for greater productivity in

higher education, have challenged the tenure system, and have called either directly or indirectly

for more assessment at the collegiate level.

I think that most people would agree that state higher education boards are supporters of higher

education. I know that there are persons from institutions that might dispute that. They believe

that these state higher education boards, be they coordinating or governing boards, are not sufficiently being advocates for higher education. But the public would see these boards as supporters

of higher education, and these state higher education boards are increasingly calling for ways to

ensure quality and to assess student learning.

132
The regional accrediting groups would clearly be seen as supporters of higher education. You

might debate the intensity or depth of their emphasis on results and outcomes, but clearly the regional accrediting groups have called for more outcome measures of which a central one is the

measurement of student achievement.

The message from the business leadership seems quite clear. Take for example the policy statement on higher education by the Virginia Chamber of Commerce. The Virginia business leader-

& ship rallied behind a campaign spurred by John T. Hazel, a Northern Virginia business and civic

leader, who called on legislative candidates to take a pledge in support of higher education. A

large number did so. The Virginia Chamber of Commerce endorsed a number of goals about

higher education, and in the first goal it called for “focusing on student needs for knowledge and

skills by rigorously assessing results. ”

Let me modestly suggest that the Southern Regional Education Board’s Commission for Educational Quality summarized very succinctly what I believe is the consensus opinion of persons

who think about higher education and its role in America’s future. The SREB Commission said,

“Higher education is America’s number one asset. This is an uncertain world. The greater the uncertainty about the world and what America may become, the more certain we are of higher education’s role in our future prosperity. We know of the problems in our colleges and universities,

but we also know of their strengths. The problems we face are twofold. First, state and national

leaders do not sufficiently recognize the value of higher~ducation in an uncertain world. The

budget decisions are proof that higher education’s priority is slipping. Secondly, colleges and

universities do not sufficiently recognize the need to mdce changes that will keep higher education the number one asset of this nation of free men and women. In a changing nation and world,

higher education is changing too slowly.” One of the ways that the SREB Commission believes

that higher education is changing too slowly is its emphasis on assessing and reporting its results.

I think the evidence is abundantly clear that both the critics and supporters of higher education

will make the need for more effective assessment a part of either their criticism of, or a condition

of their support for, higher education.

At least three other specific issues will also lead persons outside of higher education to call for

more postsecondary assessment. One of these is remedial education in colleges and universities.

I don’t have to tell this group that the political climate is running in favor of eliminating remedial/ developmental courses in colleges and universities. And I don’t have to tell you that most

legislators and governors who discuss the subject cannot fathom how it is that a college uses a

test to place a student into a remedial/developmental course and then does not require that student to pass the same test to exit from remedial/developmental studies. I look for state policy -

makers to insist on a more consistent and systematic evaluation of remedial programs.

The increased use of technology aid distance learning will also increase the emphasis on assessment at the postsecondary level. Whenever distance learning and technology is discussed, either

the first or second question asked has to do with the quality and integrity of these programs.

Elected officials want to know, “Do these programs work?” The way they generally want to

evaluate them is “Do students enrolled in distance learning programs do as well-on tests-as

students in regular courses? If you can’t answer that question generally in the affirmative, you

will be out of luck in the distance learning business. Either way, distance learning and the use of

technology in education will lead to more systematic assessment at the college level.

133
>.,

J.

I don’t know the effect of a small but growing effort to use college-entry placement tests with

high school juniors. We now have, on a statewide basis, states that are making available the college-placement mathematics or writing tests for use with high school juniors to influence their

decisions about the courses they take as high school seniors. I don’t know where this will lead us

in terms of college-level testing, but it will call more attention to this placement testing and draw

attention to the fact that some institutions have a rising-junior assessment as well as entry-level

testing.

.*

Finally, let me underscore my view that assessment must answer questions asked by the public.

Legislatures and state higher education boards of lay citizens are less prone than educators to “let

the perfect drive out the good” (or marginally good...or possibly good). The argument that a collegiate assessment system doesn’t capture all that is important or measure vital intangible factors

in higher education will not be seen as a sufficient reason for doing no assessment. Some have

said that legislators, for example, don’t make policy by relying on precise information and strategies that are limited and tightly focused. To put it in more descriptive terms, they often don’t

throw darts when making policies—they throw hand grenades. When you are throwing hand grenades, getting close can be good enough. A good college-level assessment, not a perfect or nearperfect assessment, will be close enough for lay citizens who are looking for answers to basic

questions that higher education is being asked by the public.

I don’t see in the political climate today a tidal wave of ernpha5is on postsecondary assessment. I

do see a steadily rising interest and a greater emphasis on assessing what college students know

and can do as well as more efforts to assess the effects of higher education on students’ skills in

communications, computation, and problem solving.

134
THE VIRGINIA EXPERIENCE

Margaret A. Miller

State Council of Higher Education for Virginia

Virginia was the first state to establish state-mandated, campus-based assessment. In 1986, the

.< State Council of Higher Education for Virginia and the Virginia General Assembly thought that

this approach to assessment could meet the dual goals of program improvement and accountability. Of the two, however, improvement took priority, which influenced a number of decisions

about the shape of the program.

. Virginia would not implement state-wide testing, the existing model for assessment at that

time, and assessment would be not j ust campus-specific but program-specific. This decision

was based on the assumption that only when results were tied to specific curricular goals

would faculty use the results for program improvement.

● Faculty would own assessment, so that they would take its results seriously. This meant that

those faculty responsible for programs would choose, tailor, or create the means of measuring

progress towards their curricular goals. State-level oversight would focus on whether results

were produced through reasonable and sound methodolog~es, analyzed, and used to guide appropriate curricular action.

. Finally, in order to encourage faculty and institutional commitment to campus-based assessment, the program was designed to make cross-institutional or even cross-program comparisons impossible.

Improvement

The original assumption that a program-based approach to assessment can improve programs

turned out to be correct, according to the Joint Audit and Review Committee (JLARC) of the Virginia General Assembly. JLARC concluded, in a recent review of the Council’s work, that assessment in Virginia “appears to result in significant curricular reform” (1995 Senate Document

No. 36).

The use of assessment results has been chiefly at the level of the majors: there are many examples of minor to major curricular revisions, revisions of course sequencing or requirements, an increased emphasis on certain skills, changed admissions requirements, the introduction of

integrative experiences (for purposes of assessment, but nevertheless a pedagogically sound

move), and a developing tendency to see the major as “an overall entity and not simply a random

collection of individual offerings taught by, and under the control of, individual instructors” (a

comment from one institution’s restructuring report).

Moreover, as Virginia’s institutions have struggled with the 1994 General Assembly’s mandate

to restructure their operations to “effect long-term changes in the deployment of faculty, to ensure the effectiveness of academic offerings, to minimize administrative and instructional costs,

[and] to prepare for the demands of enrollment increases,” assessment has taken on a new function. In the first eight years of the program, it was unusual for a campus to use assessment results

for purposes of planning or resource allocation. However, when asked to evaluate the effects of

135
>..

d.

restructuring, most institutions said that they would rely on their assessment programs for information about whether the changes were being made at the expense of learning. And assessment

will have another critical role to play in restructuring, insofar as institutions are developing ways

to award credit not on the basis of seat time but of demonstrated competencies. The maturity of

assessment has provided the institutions with the evaluation measures needed to undertake such

experiments responsibly and creditably.

So far, however, the assessment process itself has had its most profound curricular effect on gen-

.= eral education, paradoxically because the process usually broke down when institutions tried to

determine what their students had learned in the courses taken outside the major. This failure revealed a general lack of agreement about what students were supposed to learn from the required

courses. As one institutional report put it, “The distribution requirements are not organized

around a set of pedagogical principles; they are designed to ensure that each relevant department

is represented in the requirements. ” Since the implementation of assessment, virtually every public institution in Virginia has revised its general-education program. resulting in programs organized around learning goals rather than departmental boundaries.

But it remains to be seen whether the new curricula will be any more susceptible to assessment

than the old ones were. Some institutions have built promising assessment measures into their

new curricula, but others have postponed that task. And what even successful assessments of the

revised general education programs will not tell us is whether Virginia’s students, across the colleges, are developing to a satisfactory degree a few fundamental intellectual skills through the undergraduate years—for instance, to think critically, communicate effectively, and solve problems.

Accountability

Since campus-based assessment has fulfilled its promise to improve programs, it has met one accountability need: public higher education in the state can assure the public that it is engaged in

continuous improvement. But the sanguine belief that program-specific, non-comparable results

would satisfi public information needs turned out to be unfounded, since they are provided without the larger context that might give that information meaning to those outside the program. Besides its decontextualized nature, the volume of information that program-based assessment

generates has been also a problem: for the first eight years, assessment information was delivered

biennially to the council in documents best measured in inches. And sheer bulk aside, the information on student accomplishment that they contained has been difficult to decipher, even for

those dedicated to the task. On the basis of those reports no one, presumably not even the writers, could answer the global questions the public and policy-makers actually ask, like “How well

is this program (or institution, or the higher education system in the state) doing?” Nor does incremental, piecemeal change—the results of program-based assessment when it works—make

for a coherent story.

Various constituencies of higher education need information about its results for varying purposes. For instance, students and their parents need information in order to make wise enrollment choices, while policy-makers need it in order to make decisions about planning and

resource distribution. In order to meet those needs, rather than reform the original assessment

136
,.

program to make it do a job it was not equipped to do, the Council has proposed and is beginning to develop a series of publications designed to highlight some indirect indicators of institutional performance, such as progression and graduation rates. This series of publications is

designed to answer the questions policy-makers and the public actually have: who enters the public colleges in Virginia, what are their experiences there, how well they succeed, and what happens to them after graduation.

One of the striking results of interviewing the various constituents of higher education about .-3 what they want to know is discovering the degree to which learning and cognitive development

are not central concerns for most of them. Parents, for instance, want to know what jobs students

will get when they graduate and the cost of the credential that is the necessary if not the sufficient condition of their children’s entry into the middle class. Going to college in Virginia has

gotten considerably more expensive in the past five years; tuition, always high, has gone up by

almost a third for in-state students and more than that for out~of-state ones. The steep price increase has, at its worst, generated a kind of consumerism that is manifested as a desire to know

how to buy the ticket to the best jobs at the lowest possible price. At its best, it leads to the desire

to acquire, at the lowest possible cost, the skills and knowledge to do that job, and the next, and

the next. We would like the parents in Virginia, for instance, to ask the kinds of questions that

Governor Romer, as the parent of seven children, has been posing to the higher education community. He tells about how hard it was to decide whereto segd them to college, to determine

where he would get the best return in high-quality learning on his private investment in his children’s future good.

Legislators generally want to get students through the system as quickly, smoothly, and efficiently as possible. But they too can ask better of colleges and universities than that. While to

some legislators the 1994 restructuring mandate may simply have been an attempt to reduce the

per-unit cost of educating a student, for others, the quality of that education is of paramount concern. They are receptive, in other words, to the argument that we should be aiming for learning

productivity rather than cost-cutting. And the business community reinforces that position when

it declares its need for graduates with technical skills and general intellectual skills such as communication or problem-solving, as well as the capacity to work in teams and with people from

other cultures and subcultures. But if learning productivity can be expressed as learning divided

by cost, in order to inform policy-makers about what learning is being produced at what cost, we

need to find some comparable ways to assess that learning directly.

So the Council staff is beginning to talk, in cooperation with several other SREB states, about

the possibility of administering a performance-based, general intellectual skills test to a sample

of graduates across the state. In Virginia we are willing to consider this strategy after almost a

decade of refusing to do so because there now exists a performance-based but standardized measure of such skills, Such an examination would be a third term, of which the other two are program-based assessment and indicators of institutional mission; as a direct but comparable

measure of learning, it both would fill the hole in the indicators project and could conceivably

improve programs.

137
J.

Such a testing program does, however, entail a series of challenges. First, institutions need to

agree that the examination would actually permit students to demonstrate intellectual skills that

they are committed to develop through their curricula and co-curricula. Otherwise we cannot expect them to translate the test results into curricular and co-curricular changes that make a difference.

Second, we would need to determine how much of what we anticipate will be the superior skills

of the seniors at our highly selective colleges and universities, compared to those at the virtually

r -- open-door institutions, should be attributed to what they have gained from the college experience

and how much attributed to what they brought with them.

Then, state bureaucrats would have the daunting task of explaining the difference to the public to

whom higher education is trying to be accountable. The experience of interpreting the institutional indicators may foreshadow the difficulty of that task. Despite explanations of the relationship between progression and graduation rates and the academic and demographic characteristics

of the incoming classes, for instance, the public tendency was to make simplified j udgments that

did not take those relationships into account. Rather, several unselective institutions were

thought by some to have failed because they graduated fewer students than their more selective

counterparts. To the degree that information on student accomplishment is collected and disseminated nationally, that problem will become national in scope.

In Virginia, the higher education community, with the unfores~en but very welcome support of

the business community, has been successful in convincing state officials that collegiate education is a strategic investment on the part of the government. But now, while we need to continue

to improve programs, we also need to provide the information necessary to make an informed

public and private investment, requests for which are echoed in legislatures around the country.

We hope that higher education in the Commonwealth will be able to sit securely on the threelegged stool we have constructed, whose legs are campus-based assessment, indicators of institutional performance, and a general intellectual skills test.

138
.? -

MAKING ASSESSMENT ESSENTIAL WITHOUT

LEGISLATIVE MANDATES:

THE MISSOURI EXPERIENCE

Ava L. Fajen, Charles E. Kupchella, Robert B. Stein

.: The current crisis in higher education appears, on a superficial level, to be about adequate funding. But it may more accurately reflect doubts legislators and taxpayers are having about the return they are receiving on their continuing, already substantial investment. As public pressures

for accountability increase, state Legislatures are in some cases taking matters into their own

hands (Mahtesian, 1995). Hardly a day goes by without another call for an aggressive reassessment of higher education to make it more cost effective; some even assert that higher education

in its present form is outmoded and should be abolished altogether or at least reengineered

(Gales, 1994).

All of this comes at a time when many institutions are perceived as proceeding without a coherent strategic plan to meet the needs of their constituents. Such institutions would receive multiple

benefits from the development of comprehensive plans for self-assessment to ensure continual

improvement in serving the interests of their stakeholders (Nyhan and Marlowe, 1995).

.

Missouri’s answer to the demand for accountability and assessment, referred to as the Funding

For Results (FFR) program, is a proactive strategy to head off any need for the imposition of legislative mandates. FFR operates at both the state level and at the level of individual public institutions; it seeks to balance statewide needs with institutional goals by rewarding an institution’s

success in contributing to both (Stein and Fajen, 1995).

The extremely decentralized and diverse nature of Missouri’s system of higher education brings

a special challenge to the design and implementation of statewide assessment initiatives. Missouri is home to over 400 postsecondary institutions, including 12 public community colleges

and 10 public four-year institutions. While the Missouri Coordinating Board for Higher Education (CBHE) develops and promotes state policies for higher education, it has limited authority

over the public colleges and universities, each of which is run by a separate governing board. Despite its well-established tradition of local autonomy, however, Missouri has been an active

player in the national assessment movement since the mid 1980s.

Missouri’s assessment efforts were led by Northeast Missouri State University’s value-added assessment program (McClain and Krueger, 1985). Encouraged by the results achieved at Northeast, state leaders challenged all of Missouri’s public institutions to initiate assessment activities

which would evolve into campus-wide assessment programs. In the early 1990s, a statewide task

force established new goals for Missouri’s higher education system regarding efficiency, productivity, access, quality, effective governance, and accountability. As a way to document the extent

to which these goals have been achieved, public institutions submit data annually on a broad set

of performance indicators.

During this same time period, Missouri’s Coordinating Board made a commitment to develop a

statewide performance funding initiative which would integrate budget policies with assessment

activities. The CBHE first included a performance funding component in the budget recoin- mendation for its public four-year institutions in 1991 as the FY 1993 budget was developed. Public

139
2.

two-year institutions were added to the performance funding program in the budget recommendation for FY 1995. The performance funding elements that serve as a basis for the FFR budget recommendation are a selected subset of the broader set of performance indicators.

Currently, with support from the Fund for the Improvement of Postsecondary Education

(FIPSE), Missouri is expanding and refining its performance funding program. While institutions

continue to be rewarded for their contributions to the achievement of statewide goals, e.g., increasing the number of minority graduates, they are eligible for additional rewards if they design

e. and implement their own mission-based performance funding programs as a way to stimulate

and document improvements in teaching and learning on their campuses. Thus, Missouri’s current assessment policies encourage institutions to develop programs that address both accountability issues and the strengthening of institutional quality.

Missouri’s Current Assessment Activities %

In contrast to assessment programs that evolve in isolation, Missouri is working toward an integrated assessment program that drives agendas for planning and policy development at both state

and local levels. The goals of Missouri’s policies on assessment, accountability and performance

funding include: (1) enhanced understanding of and communication about assessment, accountability and performance funding issues among stakeholder groups; (2) enhancement of stateIevel strategic planning for higher education; (3) reinforcing and expanding institutional plans

for the improvement of teaching and learning; and (4) stfingth~ned legislative and public support of our public institutions.

To reach these goals, Missouri’s performance funding initiative, Funding for Results (FFR), operates at two levels. The two tiers of Missouri’s performance funding program are referred to as

state-level and campus-level. The list of state-level performance funding elements includes some

that apply to all institutions and some that are sector-specific. Although the state-level performance funding elements tend to be oriented toward productivity and efficiency, recent refinement

of the state-level portion of FFR is producing a gradual shift toward a focus on the measurement

of quality. In the beginning, Missouri used performance finding to encourage institutions to begin to assess student performance both in general education and in the major. As institutions

have increased their commitment to assessing student performance, the amount of reward for

simply doing assessment has decreased while new rewards linked to the performance of students

on particular measures have been added to the model. Missouri’s program, however, still provides institutions with a small reward simply for engaging in student performance assessments.

Decisions concerning the selection of assessment instruments remain the prerogative of institutional personnel.

When Missouri’s performance funding program first began, it did not make a lot of waves. It involved a very small percentage of the budget, utilized relatively noncontroversial state-level elements, and was proactive (not resulting from a legislative mandate). The state-level portion of

FFR, which was originally the entire program, received mixed reviews on the campuses. Some

campuses viewed it as an effective strategy for demonstrating accountability, while others questioned its effectiveness. The fact that FFR dollars are not a budget line item, but are rolled into

the general allocation for each institution, has made its fiscal impact relatively invisible to faculty at recipient institutions.

140
.’ -

In recent years, this initiative has clearly evolved and has gained greater visibility. The definition

of state-level elements has moved from simple productivity measures toward a greater emphasis

on measures of quality, the size of the performance funding budget component has increased,

and now rewards for using performance funding at the campus-level have been introduced. As a

result of these changes, awareness of the program has increased and assessment and performance

funding issues have begun to gain more attention from institutional presidents and chancellors,

key legislators, and the governor’s staff.

.-<

Activities to support the expansion of the FFR initiative have been underway for more than a

year. By encouraging institutions to incorporate performance funding principles at the campus

level, this initiative increases ownership and support at the local level. To promote statewide

ownership of the FFR project and build support among policy makers, groups representing all of

higher education’s stakeholders have been assembled. In addition, the Missouri Assessment Consortium, a grass roots organization of assessment coordinators from the public four-year institutions, has been consulted. All of these groups have met periodically to help shape Missouri’s

approach to assessment, accountability and performance funding.

In another significant piece of the FFR effort, a statewide group of faculty and students have

worked during the last year to define crucial teaching and learning issues and to find ways to

make the demonstration and reward of teaching and learning improvements a more centrai focus

on each campus. The result of their efforts has been published in booklet form and distributed to

administrators and campus-level performance funding teams at all public institutions. Taken together, FFR activities during the last year+xtensive efforts to give all constituencies a voice in

the process, unprecedented statewide communication efforts, and deliberate measures to keep

teaching and learning issues in the spotlight—have ail focused on promoting institutional success in creating measurable improvements in teaching and learning.

Missouri’s governor and legislature have evidenced comfort in using performance funding in the

allocation process without the heavy hand of a legislative mandate. This initiative is inten- tionally structured so that it will not drastically affect the overall budget of an institution in any

given year. The following table provides both recommended and allocated amounts of the

budget from FY 1993 forward for public two- and four-year institutions.

141
.-

Requested vs. Appropriated Dollars and Percentages

Fiscal Dollars

Year Requested

1993

,--- 1994

1995

1996

1997

Fiscal

—

.5 million

.7 million

.5 million

Dollars

Year Requested

1993 6.8 million

1994 6.7 million

1995 10.0 million

1996 13.2 million

1997 13.5 million

Public Two-year Institutions

Percent* Dollars

Requested Appropriated

1.5 500,000

1.8 799,690

2.9 ****

Percent

Appropriated

.6

.9

Public Four-year Institutions

Percent* Dollars Percent*

Requested Appropriated Appropriated

1 0 0

1 ‘3,027,849 .5

2 4,272,131 .8

2.2 7,010,000 1.2

2.4 ** **

* Percent of total core general revenue operating budget

** Currently under review

While the amount of money associated with performance funding remains relatively small, i.e.,

less than 3 percent of the total state allocation for either sector, it is large enough to have an impact. For FY 1997, more than $16 million are included in the performance funding portions of

the CBHE budget recommendation for higher education. This represents approximately 19 percent of the new money recommended for four-year institutions and 13 percent of the new money

recommended for two-year institutions. Each year has seen a slight increase in the amount recommended as well as the amount allocated. The message to institutions is clear; performance funding in Missouri is on the rise.

Campus-Level Performance Funding Projects. The campus-level tier of FFR is in its pilot

year. Each campus was eligible to be recommended for performance funding rewards in the next

budget cycle if it submitted a plan for a pilot FFR project that identified a major teaching/learning issue. Although the submission of a campus-level pilot project for the 1995/1996 year was

voluntary, all 29 public institutions chose to participate. The proposals for the pilot projects were

reviewed by statewide groups of institutional representatives and by CBHE staff,

142
.,.

The submitted projects reflect differences in focus and scope. but all address specific teaching/learning issues. Each project has the support of the institution’s president and/or chancellor,

involves a campus-level FFR team composed of both administrators and faculty, and links a portion of the institution’s operating budget to rewarding performance. All projects involve the use

of defined intervention strategies, the collection of baseline data, and an evaluation plan.

The goals of the proposals range from improving the basic study skills of community college students to enhancing the residence experience of doctoral students. While some projects directly ,. &

address ways to improve the performance of students on assessments, others focus on enhancements for student advising or on easing the transition from the community college environment

to the four-year institution.

In a few cases, institutions have chosen to develop an open process whereby all units are invited

to submit proposals that address a teaching/learning issue within the mission framework of the institution. This approach has the advantage of integrating performance funding into an ongoing

strategic planning process. In such instances, Funding for Results is not perceived as a new and

separate initiative; rather it is used to re-energize an existing infrastructure. Initial reports suggest

that this approach has been very successful: many more proposals are being submitted than can

be funded.

State policy makers are beginning to perceive Funding for R~sults as a catalyst for change. On

some campuses, faculty are perceiving renewed support for their teaching roles, and an increased

level of interaction with their administrators. Although demonstrating a rich diversity of approaches, each of the campus-level projects now underway reflects a significant institutional

commitment to the improvement of teaching and learning, and reflects an understanding of the

importance of using data as a basis for decision making and policy development.

An Institutional Perspective: Southeast Missouri State University

The experimental development of a campus-level Funding for Results (FFR) program came at a

very opportune time for Southeast Missouri State University (SEMSU). The campus was involved in a comprehensive strategic planning process as the campus-level FFR idea began to

take shape. Strategic plans were under development within each academic unit and at all levels

within the organization. One of six strategic priorities identified early in the process was “Offer a

top-quality curriculum with a solid liberal education as a foundation for preparing graduates for

leadership positions in society.” Another priority focused on “enhancing learning.” Each unit

was asked to indicate how it would go about addressing the university’s overall priorities. Each

academic unit was also asked to assess the general effectiveness of its curricula and to indicate

how this effectiveness is/will be determined on an ongoing basis.

As the University Planning Committee worked to develop campus “buy-in” of the strategic planning process, it became clear that one of the historical sources of skepticism about the effectiveness of planning on the campus was that planning never seemed to be connected to budgeting.

The University Planning Committee realized that if planning were to become an effective way of

life on the campus, faculty would need to know that planning and budgeting are indeed connected. The timing of the FFR program afforded the campus an opportunity to build a planningbudgeting connection into the early stages of its major strategic planning initiative.

A second aspect of the timeliness of FFR had to do with assessment. The campus received approval of its institutional assessment plan from its regional accrediting body in the spring of

143
),

1995 and the campus was moving through the second phase of refinement of its assessment program. At the time campus-level FFR began to take form, SEMSU was beginning to connect strategic planning, program assessment, and continuous quality improvement. FFR was seen as a

way to accelerate the use of assessment as a means of driving continuous improvement.

SEMSU’S approach to the state challenge of developing and implementing a campus-based FFR

pilot program was to design a process whereby all units could have an opportunity to get involved. As a result of a team effort, faculty, departments, and other units were invited to offer

.: -- their strategic objectives having to do with teaching and learning enhancement for possible funding. The pilot program was funded by reallocating a portion of the institutional/campus budget.

The review and funding of FFR initiatives was scheduled to occur immediately after the units

had completed their strategic plans.

The program was designed and the reviews were carried out by a “Funding for Results Team.”

This fourteen-person team is made up of members of both the University Planning Committee

and the University Budget Review Committee, providing yet another connection between budgeting and planning. The review was set up as a grant proposal competition using a simple format

with a two-page limit for the narrative. Clear criteria for evaluation were spelled out by the FFR

team. These included 1) degree of direct focus on improvements in teaching and learning, 2) clarity of definition and measurability of outcomes, and 3) clarity of the relationship of the project to

the unit’s strategic plan.

Although expectations were high, SEMSU’S campus-based FFR team was overwhelmed to receive sixty-nine proposals. These included a wide variety of proposed projects seeking to enhance oral communication skills, written communication skills, technological communication

skills, problem-solving skills, critical thinking skills, and student success in general. Some examples will help illustrate the variety and scope of the projects. The Department of Speech Communication and Theater proposed to develop and test oral communication modules for selected

courses in other majors. Improved speech communication would become one of the curricular

objectives in the selected courses. Gain would be assessed subjectively by the course instructors

and by the students, themselves, through course evaluations. The principal assessment measure,

however, would bean instrument developed by the Speech Department. Ultimately, this “module” idea would be developed as a “speaking across the curriculum” program, university wide.

The Departments of Criminal Justice, Industrial Technology, and English, (and several other departments) proposed projects to enhance student writing. The Department of Criminal Justice

proposed to identifi students likely to have difficulty passing the university’s writing proficiency

exam and direct them to a special writing workshop. Industrial Technology proposed a series of

workshops to help faculty teach writing about disciplinary subjects more effectively. This project’s effectiveness would be assessed by measuring the number and richness of writing assignments by targeted faculty before and after the workshop. The Department of English proposed to

give faculty members computer upgrades equivalent to computers in a new “computer composition” laboratory. Effectiveness would be assessed via number and type of composition lab assignments given to students by faculty. Currently, SEMSU has a university-wide writing assessment

program. Students are required to pass a writing exam (with a highly developed, standardized

scoring rubric) after completing 75 hours of course work. All of the FFR writing enhancement pilot projects plan to use baseline and outcome data collected by the university’s writing assessment program.

144
--

.=

.. .

Four departments proposed programs to develop student technological communication skills.

The development of skills such as the effective use of multimedia in making presentations would

be assessed by asking the students to demonstrate the skills. Several departments proposed projects that would involve students in “real-world” problem-solving activities to enhance critical

thinking skills. Industrial Technology seniors would be given problems actually being experienced in regional manufacturing industries. The students, in small groups with a faculty mentor,

would tackle the problem and propose a solution. This learning strategy was presented as an “ulti-

.* mate” test of the student’s ability to solve problems as members of a work team. Sociology and

Anthropology, Biology, Criminal Justice, Psychology, and other departments proposed quality

improvements in internship programs with similar intentions of engaging students in real world

problem solving situations.

There were many proposals to enhance student success ranging from instituting daily quizzes in

key business courses to providing various forms of tutorials for students in math, accounting,

and other “difficult” courses. The success of the interventions would be measured against historical baseline data or, in some cases, “control” classes not receiving the intervention. Also proposed were a large number of teaching enhancements mostly having to do with the use of

information technology. There were twenty-three proposals asking for computers or computer-related hardware and software. Reviews are still in progress at this time; the FFR Team has been

impressed by the breadth and depth of ideas that have been ggnerated.

Although budget constraints will preclude the funding of all worthy projects, it is clear that as a

result of our FFR pilot project many exciting ideas are germinating on campus. Budget decisions, not just involving our FFR allocation, but those that are part of the larger budget process

on campus will be influenced. SEMSU is succeeding in using assessment to make improvements. In addition, reward structures are being realigned so that faculty are more integrally involved in shaping the institution’s future. Teaching/learning issues are becoming a central focus

in the budget and planning process of the university.

State-Level Perspective on Funding for Results

From the perspective of state policy makers, Funding for Results has elevated the discussion

about the needs of higher education from a litany of complaints concerning a lack of resources to

a focus on performance and institutional achievements. While resources remain a major concern,

conversations about opportunities, constraints and accomplishments are moving toward a datadriven rather than an anecdotal basis.

Requiring institutions to systematically identify and analyze outcomes across a wide array of elements has a number of positive effects. Institutions are forced to engage in self-reflection about

effectiveness and value using agreed-upon criteria. Baseline data are established so that improvements can be tracked over time. When visioning for the future, institutions are pressed to identify

measurements that will clearly demonstrate achievement of specific goals. Institutions are also

better positioned to ensure continual improvement by systematically using assessment information to evaluate their progress. Last but certainly not least, as accomplishments are documented,

higher education’s communication with external publics can be greatly enhanced.

As an initiative for change, performance funding helps to focus attention on assessment. By linking a

tions

small portion of budgets to results at both the state and campus levels, i.e., state appropriate institutions, and internal appropriations within institutions, both state level priorities and

145
J

institutional aspirations are made salient. A complex assessment program can be made to serve

multiple purposes, i.e., accountability to external publics and improvement of teaching/learning

on campus, simultaneously. When linked with assessment and accountability activities, performance funding can help to narrow the difference between faculty perspectives and those of state

policy makers.

At the state level, the distribution of a limited pool of funds among the higher education institutions is being reexamined. Institutions contribute differentially to state priorities and should be re-

.,. ~- warded accordingly. A key issue then is the setting of state-level priorities and how assessment

results should be used to demonstrate institutional responsiveness to state interests. In addition,

the total amount of money allocated by means of performance funding should remain small

enough that changes in institutional allocations from one year to the next are not too great.

At the campus level, the majority of funds within colleges and tiniversities are committed years

in advance, e.g., personnel costs. The expenditure of contingency money, therefore, takes on extra symbolic significance. Traditionally the distribution of any flexible portion of university and

college budgets has been based on historical precedent. The use of a portion of the budget to reward results thus requires a paradigm shift.

Suddenly questions about what to reward, when to reward, and who should receive a reward become important considerations. Initial approaches might definS reward structures in simplistic

terms, i.e., those most responsible for bringing about a desired change receive a reward. Merit

pay systems represent this traditional approach to performance funding. The FFR initiative, however, is resulting in a broadening of the definition of reward structures through a greater awareness and involvement by personnel across the campus.

From the outset, Missouri has been committed to building a performance funding model that will

make a positive difference in the everyday lives of faculty and students. FFR is building a statewide coalition for change that links public policies on funding, assessment and accountability

with programs to improve teaching and learning. Through its performance funding initiative,

Missouri continues to raise the stakes associated with traditional assessment activities and has ensured the centrality of assessment on the campuses of its public institutions.

References

Gales, R. “Can Colleges Be Reengineered?” Across the Board, March 1994,31 (3), 16-22.

Mahtesian, C. “Higher Ed: The No-Longer-Sacred Cow.” Governing, July 1995,8 (10), 20-26.

Nyhan R. C. and Marlowe, H. A., Jr. “Performance Measurement in the Public Sector: Challenges and Opportunities. ” Public Productivity and Management Review, Summer 1995, 18

(4), 333-348.

Stein, R. B. and Fajen, A. L. “Missouri’s Funding for Results Initiative.” In G. H. Gaither (cd.),

Assessing Performance in an Age of Accountability: Case Studies. San Francisco: JosseyBass Publishers, 1995.

146
-.

Methodological Concerns: An ETS Perspective

John Mazzeo

Educational Testing Service

As a participant in this session on Methodological Concerns for the 1995 NCES State Postsecon-

.< dary Assessment Activities Workshop, I have been given two charges. The first is to say a little

about Educational Testing Service’s (ETS) general approach to postsecondary outcomes assessment. The second is to discuss some of the lessons learned from the National Assessment of Educational Progress (NAEP) that might be relevant to the post-secondary assessment context.

Section I of this paper discusses some existing ETS assessment instruments that are appropriate

for postsecondary outcomes and the general assessment philosophy underlying the creation of

these instruments. Section II of the paper discusses some lessons learned from NAEP that may

be relevant to the postsecondary assessment context.

Section I. ETS’ general approach to postsecondary outcomes assessment and the instruments and services it currently provides

In general it is ETS’ belief that the use of any test or instrument should be one part of an overall

and ongoing assessment effort. Assessment efforts shou}d inc+ude: instructor-written classroom

tests, portfolios of student work, measures of student achievement in the major, department/program reviews, alumni follow-up and student evaluation of courses and instruction.

The choice of instrument should be determined by the kinds of information that will be most useful to faculty and administrators. Assessment is not an end in itselfi it is a way of obtaining information to help focus on and understand problems. Results should suggest actions that will lead

to the improvement of teaching and learning.

ETS offers a variety of tests, instruments and services that can be used in outcomes assessment

and institutional evaluation. These offerings are intended to supplement local measures (as well

as other external measures) and to enhance total assessment efforts.

Academic Profile-A comprehensive measure of the academic skills acquired through undergraduate generaI education courses.

The test measures: college-level reading, college-level writing, mathematics, and critical thinking in the context of material from the humanities, social sciences, and natural sciences. Scores

are reported in terms of proficiency levels (in writing, mathematics and reading/critical thinking)

and as standard scale scores in the four skills and three academic areas noted.

Tests consist entirely of multiple-choice items and two forms are available: (1) a short form (45-

minutes) which provides an overall student score and institution-level scores in the four skill and

three academic areas described, and (2) a long form (2.5 hours) which provides individual and institution-level scores for the 12 skill/area combinations. Institutions can tailor the test by adding

up to 50 locally written questions to the test.

Tasks in Critical Thinking—a performance-based measure of student abilities to use higherorder or critical thinking skills.

147
>..

.J

Using an extended response format, the tests set a problem and direct the student through various

steps necessary to solve it. Tests were designed by college faculty and were intended to resemble

what students are required to do in the classroom and the world of work. Each task is set in the

context of one of three broad academic areas (humanities, social sciences or natural sciences)

and consists of 8 or 9 short answer questions followed by an essay or report.

Scores are reported for groups (not individuals) as the percent demonstrating proficiency in each

of three skill areas—inquiry, analysis, and communication. Scoring can be done entirely by local .<--

faculty (using detailed materials developed for each test), using local faculty but directed by an

ETS consultant, or entirely by ETS.

Major Field Tests—multiple-choice tests that assess mastery of concepts, principles, and knowledge typically expected of students upon completion of undergraduate major in a given subject.

These tests are available in 14 disciplines. Score reports include a departmental summary showing total score and subscores for each student and average scores for the department. In addition,

two to eight assessment indicators (scores that relate to subfields within a major discipline) are

reported for groups of tested individuals (but not at the student level). National comparative data

are available for each test. Departments may add locally written questions that address departmental concerns.

Participants interested in more information about any oflhe above products should contact

Nancy Beck ofETSat609-921 -9000.

Section II. What lessons have been learned from NAEP?

Stakeholder consensus is a must

Consensus and stakeholder involvement are extremely important from the outset. Ideally, students, faculty, university administrators, state policy and legislative officials, the business community, and the public at large should have an opportunity to communicate their ideas and

concerns openly and aggressively as assessment systems are being established. Agreement needs

to be sought regarding: i) assessment purposes, ii) the identification of pertinent skills/knowledge to be assessed, iii) assessment formats, iv) survey designs, v) performance standards, and

vi) reporting formats.

Our experience with state NAEP suggests to us that a major contributor to the success of state

NAEP has been keeping state assessment officers and curriculum staff actively involved in the

design and conduct of NAEP. The curriculum frameworks on which the assessments are based

have been developed (under NAGB and CCSSO auspices) using a concensus process in which

state curriculum experts have had opportunity to actively contribute. The NAEP instruments constructed from those frameworks are reviewed (at the field test and final form stages) by designated state curriculum staff and their input discussed and, where appropriate, incorporated.

Regular meetings of “THE NETWORK” (which consists of state assessment directors and curriculum staff) are held (3 times a year) and provide an opportunity for state input on all aspects

of the program (minimum participation guidelines, administration and monitoring procedures, reporting formats, plans for press releases).

Of course, talking about getting concensus and actually obtaining consensus (particularly with respect to assessment frameworks and instruments) are two different things. There are a variety of

things that can go wrong. One way in which concensus can backfire is in producing an instru148
ment that represents the lowest common denominator among the constituencies. Another problem, (more often the case in our experience with NAEP) has been a tendency for panels with diverse viewpoints to produce an instrument that is overly ambitious. Getting broad based panels

to acknowledge the realities of cost constraints and administrative realities is not always easy. At

a minimum, the concensus process needs to be iterative involving the preparation of multiple

drafts of planning documents with mechanisms after each draft for eliciting and incorporating

stakeholder comments.

.<

Assessments must be perceived as improvement oriented

Our NAEP experience is consistent with the view that the greatest likelihood of success will

acrue to assessment efforts that are seen as improvement oriented. Institutional performance data

should be accompanied by contextual data about the student populations being assessed, their opportunities to learn the assessment materials, and the resources available to faculties and institutions in support of learning. Reducing the emphasis on the “horse race” (i.e., comparisons across

institutions, or departments within the institution) and focusing more on institutional improvement overtime is in the best interests of everyone involved.

In my opinion, NAEP’s pre-eminent purpose should be the reporting of credible trend data. It

provides one mechanism for states and the nation to monitor their movement toward instructional improvements and achieving of educational goals. Tre~d results are treated prominently in

both national and state reports.

Furthermore, NAEP collects background data on student demographic characteristics, student

self-reports on instructional experiences, study habits, home support for schooling activites;

teacher reports on their levels of schooling, certification, professional development, and teaching

activities; school reports on programs, policies, and school climate variables. An attempt is made

to focus attention on interpreting performance data (and any changes or improvements in performance) in light of these background variables. Such data also allow for identifying potential

areas of improvement in the delivery of educational services.

Perhaps the most useful document for states that participate in NAEP is an individualized report

on their states’ test results and the background data collected. While each state’s report does contain sections comparing the state-level test performance to that of the nation, a large portion of

the report is focussed on what instructional activities are going on in the classroom, the training

and preparation of the teachers delivering instruction, what students are doing with their time outside of school, and home support activities for learning. My experience in talking to state testing

directors is that this latter information is far more useful to them in talking to schools, policy analysts, and legislatures within the state about how education might be improved.

A common metric for comparing performance across jurisdictions will have appeal

Most institutions (and jurisdictions) view their goals, their curricula, the problems they face and

the populations they serve as somewhat unique. They at least initially fear cross-institution comparisons and the inappropriate (and potentially punitive) consequences associated with (what

they see as) inappropriate inferences about the quality of schooling at their institution.

Growing interest in state NAEP suggests that if a common yardstick can be demonstrated to be

credible and fair (and if the first two points above are successfully achieved), institutions may

come to view such results as highly valuable. At NAEP’s inception (over 20 years ago) state149
level results were politically unthinkable. By the late 80’s, the states themselves were requesting

specific state-level results. In 1990, the first year of state NAEP, 40 jurisdictions participated in

this voluntary program. The numbers rose to 44 for both the 1992 and 1994 assessment cycles.

The number is 50 for the 1996 state assessment (the third such assessment). Furthermore, there is

now interest on the part of some of the larger school districts around the country (Philadelphia,

Milwaukee, Atlanta, to name a few).

It seems to me that the many state educators, legislatures and policy staff have come to value the

. ---- comparative “benchmark data” (national and state comparisons) and the ongoing high quality

normative results that the program provides. The 1996 assessment will include an attempt to link

results to the TIMMS (an international science and mathematics assessment). If successful, such

linking would allow for the possibilities of comparing results to a large number of countries. In

my opinion, this has resulted in a further increase in participation in state NAEP.

Sampling versus all student testing

Unless one is testing for student certification purposes, sampling-based assessments offer considerable advantages. There are two types of sampling to distinguish here: examinee sampling and

item sampling. With examinee sampling, a typically small proportion of the school/student population is tested and the intent is to generalize the results to the entire population of schools/students. With item sampling, any particular assessed student is a~ministered only a portion of the

pool of assessment tasks. Combining these two approaches yields a matrix sample-one first

samples a group of students, divides the sample further into subgroups, and administers each subgroup a portion of the pool of assessment tasks. Matrix sampling approaches are what are used in

NAEP. Matrix sample approaches to assessment are highly efficient in that they minimize student testing burden while allowing for a broad representation of the item pool.

When confidentiality of participants within a jurisdiction (institution) is assured, sampling-based

approaches tend to minimize concerns about cheating or inappropriate kinds of “teaching to the

test. ” However, a potential problem with sampling-based approaches concerns the motivation of

the tested individuals. Why should an individual in such an assessment work hard to do their best

if there is nothing at stake for the individual? Motivational issues are particularly salient in situations where standards-based reporting is in use. Results from assessments are sometimes expressed in terms of the proportions of students whose test results exceed some pre-established

standard. However, the standard is often set without considering that portions of the assessed

population may not be trying their hardest on the test. Such considerations are also confounded

by the level of effort demanded by the task. If the tasks involve considerable student effort, the

performance of unmotivated students may look unexpectedly low.

The evidence from NAEP on the effects of motivation (or lack thereof) have been mixed. Research by O ‘Neill et al. using NAEP math tests showed that financial incentives for performance

($1 per each Conectly answered item) raised test scores by 1/2 standard deviation (sol) among 8th

graders but did not do so for 12thgraders. Work by Kiplinger and Linn, in which two sets of

NAEP items (which were administered to students under low stakes situations for the students/schools) were embedded within a high stakes state testing program. Higher performance in

the high stakes context was found for only one of the two item sets. But clearly, taken as a

whole, these studies highlight a weakness of sample-based low-stakes testing.

150
If an examinee sample-based approach is chosen, such an approach can be carried out in terms of

a cross-sectional or a longitudinal design, e.g., NAEP is a cross-sectional design. Every 4-years

different representative samples of 4th, 8th, & 12th graders are tested. The sample of 8th graders

tested in 1994 is not the same individuals who were tested in 1990 (as 4th graders). In a longitudinal design, the individuals tested in 1990 (as 4th graders) would be tested again in 1994 (as 8th

graders) and again in 1998 (as 12th graders). Each of the two sample designs has pros and cons

associated with it.

.?

Cross-sectional versus longitudinal designs

Cross-sectional designs are, in some sense, simpler to deal with. An individual is tested once and

no follow-up is required. The cross-sectional design is especially amenable to a group-level focus describing the aggregate achievement of the current population and, as such, efficient designs (like matrix samples) work well within this context. There are, however, some clear

limitations. It is often tempting to interpret across cohort comparisons (e.g., 4th and 8th graders

at a given time point of 8th graders in 1994 vs 4th graders in 1990) as showing the amount of

growth in test score performance over the indicated period. But such interpretations are confounded by other factors. Comparing 8th graders and 4th graders in a given assessment year confounds cohort differences (in, say demographics and SES) with growth. Comparisons of the

same cohort (e.g., 12th graders in 1994 with 8th graders in 1990) ignores the fact that drop-out

occurs and the drop-out rate can be expected to be correlated With level of achievement. Another

limitation of the cross-sectional design is that it is next to impossible to establish convincing

“causal relationships” between process and product variables. In cross-sectional surveys, correlational patterns can be observed and established but “correlation does not imply causation. ”

Longitudinal designs can be difficult to deal within certain contexts since the individuals initially tested must be traced and followed up with later testing. (This may be less difficult in the

postsecondary school context than in other settings but will probably still cause problems.) In a

longitudinal design, the emphasis is on getting reliable individual measures of growth and aggregating these to describe growth at the group level. This generally involves tests that are reliable

at the level of individual scores and hence such designs are not very amenable to the efficient matrix sample designs discussed earlier. In particular, domain coverage can be a problem unless

each individual is tested for a fairly long period of time. There are, however, some clear advantages to these designs. The initial testing and follow-up of the same individuals and the presence

of individual student scores make the provision of student feedback possible. This may have desirable effects on increasing student motivation. Furthermore, if growth is the focus, the cohort

effects which confound interpretations in a cross-sectional design are mitigated (since growth

measures at the level of the individual are obtained). If one is interested in relating process and

product variables, time sequencing_ information (and a variety of well developed data analytic

techniques) make this easier to do in a longitudinal design.

Some Concluding Remarks

In concluding my comments for this session, I feel it only fair to note a few things about my

background since they no doubt color my views on the feasibility of broad-based postsecondary

outcomes assessment. My training and background is in the area of psychometrics and educational statistics. Throughout my career, my role on the various assessment projects that I have

been involved with has been primarily that of methodologist and data analyst. I am not a test de151
veloper (i.e., I do not write items), nor am I involved in educational administration or policy

analysis (i.e., I have relatively little to do with the politics of educational assessment activities).

None of the projects I have worked on have involved postsecondary assessment. Therefore, I

hope the session participants will excuse my naivete’ as to the peculiar assessment challenges

that arise in the postsecondary arena and my focus on methodological rather than substantive

concerns.

In recent years, I have worked almost entirely on the State Assessment component of NAEP. As

most (if not all) of you know, NAEP is a sample-based assessment that is designed to provide

group-level information on the educational achievement of America’s in-school student populations at grades 4, 8, and 12. Beginning in 1990, group-level data has been provided on a state-bystate basis to jurisdictions that participate on a voluntary basis. In my view, NAEP is an

extremely important educational indicator and the conduct of a voluntary state NAEP has been

successfully accomplished. In short, I am a believer in the feasibility and utility of centralized,

standardized, sample-based assessments like NAEP for the monitoring of educational progress

and think that this model represents an attractive alternative for cross-state assessments of postsecondary outcomes assessment.

I am quite grateful for having had the opportunity to work on NAEP. During my tenure of involvement with NAEP it has been carried out in an atrnosph~re of general political support and

with a great deal of economic resources available. Because of this environment, NAEP has had

the luxury to design and execute a broad-based cross-state assessment project with a great deal

of scientific rigor using what I believe to be cutting-edge assessment, methodological and data

analytic methods. What I have learned about carrying out such assessments I have learned in this

ideal environment. As a result, my views on the prospects of the NAEP model for postsecondary

assessment are no doubt optimistic—many of you no doubt j udge them as overly optimistic.

References

Kiplinger, V.L. & Linn, R.L. (1993) Raising the stakes of test administration: The impact of student performance on NAEP. CRESST Report No. 360, Los Angeles: The Regents of the

University of California.

O’Neill, H.F. (1992) Experimental Studies on Motivation and NAEP Test Performance. Draft

CRESST Report.

“U.S. Governmam Prlmlng Olficw 199S — 421-259

152