Understanding abilities, learning and development through college outcome studies: What can we expect from higher education assessment?

This document presents elements of a symposium developed by the Alverno College, Wisconsin, research and evaluation staff on the extent to which educational research that is internally driven by institutionally defined purposes and questions is valid and useful to the educational research community as a way of addiessing its larger questions about learning and teaching. The document contains outlines and discussions of symposium questions, symposium objectives potential beneficiaries from advances on the questions posed by the symposium, questions posed for Alverno College, lessons learned and unresolved issues. Also included are five research examples from Alverno College's research and evaluation department. Those examples are: a longitudinal analysis of cognitive, moral and ego develcpment trajectories; an analysis of self-sustained learniAg and development; a study of career trajectories of women; research toward a taxonomy of alumnae generic and professional abilities; and an evaluation of professional abilities and student outcomes. Appendixes contain a description of the current student body at Alverno, a description of the longitudinal sample, charts of educational and developmental stages and relationships, a summary of statistics, a summary of generic abilities, an example of coding analysis of an essay, and 114 references, (L7B) *********************************************************************** Reproductions supplied by EDRS are the best that can be made from the original document. **************************u******************************************** UNDERSTANDING ABILITIES, LEARNING AND DEVELOPMENT THROUGH COLLEGE OUTCOMES STUDIES: WHAT CAN WE EXPECT FROM HIGHER EDUCATION ASSESSMENT? Marcia Mentkowski, Glen Rogers, Deborah Deemer, Tamar Ben-Ur, Judy Reisetter, William Rickards, Mary Talbott "PERMISSION TO REPRODUCE THIS MATERIAL HAS BEEN GRANTED BY ___Maraa-Me.ntkaw-alsi_ TO THE EDUCATIONAL RESOURCES \ INFORMATION CENTER (ERIC). " U.S. DEPANTNENT OF EDUCATION OFIce or Educational Research end Improvement EDUCATIONAL RESOURCES INFORMATION CENTER (ERIC) is document has bun reproduced u ficilVed from the person Or ONPIONIIIon originating a 0 Minor chimps hue bun made to ernfX0V reproduction Points of tau or opinions stated n thedoor ment do noi necessarily represent official OEFff position or pohcy Office of Research and Evaluation Alverno College Milwaukee, Wisconsin Symposium presented at the Annual Meeting of the American Educational Research Association, Chicago April 5, 1991 BEST COPY AVAILABLE OFFICE OF RESEARCH AND EVALUATION STAFF Marcia Mentkowski, PhD, Director of Research and Evaluation Professor of Psychology Glen Rogers, PhD, Research Associate Deborah Deemer, PhD, Research Associate William Rickards, PhD, Research Associate Tamar Ben-Ur, MEd, Research Analyst/Data Manager Judy Reisetter, MA, Research Analyst/Manager Mary Talbott, MEd, Research Analyst Kathleen Schwan, MA, Research Analyst Beverly Weeden, MA, Office Coordinator Lynn Chabot-Long, Word Processor RESEARCH AND EVALUATION COMMITTEE Marcia Mentkowski, PhD, Chair Director of Research and Evaluation Professor of Psychology Zita Allen, MSN Associate Professor of Nursing Lucy Cromwell, PhD Professor of English Mary Diez, PhD Professor of Education For further information write: Office of Research and Evaluation Alverno College 3401 South 39th Street Milwaukee, Wisconsin 53215 (414) 382-6263 Austin Doherty, PhD Vice-President for Academic Affairs and Dean of the College Georgine Loacker, PhD Professor of English Kathleen O'Brien, PhD Associate Professor of Business and Management Timothy Riordan, PhD Professor of Philosophy Stephen Sharkey, PhD Associate Professor of Social Science L0 Copyright 1991. Alverno College Productions, Milwaukee, Wisconsin. All rights reserved under U. S. , International and Universal Copyright Conventions. Reproduction in part or whole by any method is prohibited by law. UNDERSTANDING ABILITIES, LEARNIM AND DEVELOPMENT THROUGH COLLEGE OUTCOMES STUDIES: 1 WHAT CAN WE EXPECT FROM HIGHER EDUCATION ASSESSMENT? Marcia Mentkaseki, Glen Rogers, Deborah Ceemer, Tamar Ben-Ur Judy Reisetter, William Rickards, Mary Talbott Office of Researdh and Evaluation ALVERM COLLEGE This symposium examines the extent to which educational research that is internally driven, by institutionally defined purposes and questions, is valid and useful to the educational research community, as a way of addressing larger questions about teaching and learning. Recent state mandates for higher education assessment have generated a spate of studies within colleges and universities designed to collect information on college outcomes. What can we expect from these studies in higher education assessment? As educational researchers invested in our discipline or as institutional representatives doing assessment, we expect that assessment -carried out by and for an institution -will improve educational practice and studeLt learning. But to what degree will we realize these expectations? The essessment movement currently is concerned with one aspect of this question, "TO what degree can assessment benefit the institution that initiates it? " This question is of primary importance to those of us doing assessment because many of us are not satisfied what has been going on in the name of assessment. In Assessment for EXcellence, Astin (1991) argues that "although a great deal of assessment activity goes on in America's colleges and universities, much of it is of very little benefit to either students, faculty, administrators, or institutions (p. ix)". Nevertheless, developing assessment practice designed for a particular institution's purposes to enhance educational benefits is clearly an important goal (Mentkowski, in press). 1 Symposium presented at the American Educational Research Association, Chicago, April 5, 1991. The authors acknowledge Joan Stark, University of Michigan, and Jon Wergin, Virginia Commonwealth University, for their role in this Division J (Postsecondary Education) symposium, and for their review and critique of this paper. Stark, who is Director of the National Center for Research to Improve Postsecondary Teaching and Learning, set the context for the symposium and served as discussant. Wergin, who is Vice President of Division I (Education in the Professions) and a member of the Alverno Research and Evaluation Advisory Council, also served as discussant. Alverno's Research and Evaluation Committee critiqued this paper: members are Zita Allen, Lucy Cromwell, Mary Diez, Austin Doherty, Georgine Loacker, Kathleen O'Brien, Timothy Riordan and Stephen Sharkey. Kathleen Schwan, Beverly Weeden, Lynn Chabot-Long and Rene Sisouphone contributed to the production of the paper. UNDERSTANDING ABILITIES, LEARNING AND DEVELOPMENT 1991 AERA SYMPOSIUM 2 Therefore, elucational researchers doing assessment are asking (see Figure 1): (1) To what degree do college outcomes studies contribute to an institution's purposes? Tbday's discussion will deal indirectly with this question, as it is incorporated in the first objective of this symposium. Tbday's discussion is designed to enable the AERA Division J membership to begin exploring this objective: (a) Tb discuss hail AERA's Divisica J can best provide a forum and critique for such studies so as to influence the direction and quality of higher education assessment as an emerging field. The presenters at this table believe AERA can and does provide an excellent forum for this discussion. (In fact, Alverno faculty and research staff made 19 presentations at AERA from 1980 to 1990.) It is our pszctice to invite critique along the way, as we are designing and implementing various research and evaluation strategies, and after sets of results are in and interpretations made. Clearly, our research and evaluation team is a primary beneficiary from today's critique, as are other institutions and their representatives who have similar concerns (see Figure 1). But what should be the nature of the critique? What criteria should such studies aim to meet? By what evidence should such studies be judged? What are the standards to which such studies should be held? Because we, as AERA-J members, are concerned with these questions, um are invested parties in today's critique. Because the higher education assessment movement has created a new context for conducting educational research, there is a leadership role for AERA-J in ensuring the quality of higher education assessment. Many educational psychologists are involved in carrying out college outcomes studies in their own institutions. These studies represent a new source of hard money funding for educational research, badly in need of research dollars. However, such studies can potentially make other comtributions in addition to examining an individual institution's quality, effectiveness, and validity. For educational psychologists, it seems unwise bo overlook higher education assessment as a source for contributions to the philosophy, principles and practice of educational research. Consequently, we ask: (2) To what degree can college outcomes studies yfeld nod knowledge re research/evaluation/measurement methods and practice, and illuminate its philosophical assumptions and values? A corresponding symposium objective is: (b) to critique the value of diverse methods for contributions to educational research philosophy/principles/practice; 1991 AERA SYMPOSIUM April 5, 1991 Figure 1. Symposium questions, objectives and potential beneficiaries. Symposium Questions Objectives Potential Beneficiaries 1. To what degree can college outcomes studies contribute to an institution's purposes? a. To discuss how AERA's Division J can best provide a forum and critique for such studies so as to influence the direction and quality of higher education assessment as an emerging field. Institutions and their representatives engaged in institutional assessment. AERA Division