DOCUMENT RESUME
TM 025 921

ED 402 338
AUTHORTITLE
INSTITUTIONPUB DATENOTEAVAILABLE FROM

Cashin, William E.
Student Ratings of Teaching: The Research Revisited.
IDEA Paper No. 32.
Kansas State Univ., Manhattan. Center for Faculty
Evaluation and Development in Higher Education.
Sep 95
10p.; Update of "Student Ratings of Teaching: A
Summary of the Research. IDEA Paper No. 20," see ED
302 567.
Center for Faculty Evaluation and Development, Kansas
State University, 1615 Anderson Avenue, Manhattan, KS
66502-4073 (individual copies, $2; complete set of
IDEA papers, $15; 20-49 copies of the same paper,$0.30; 50-99 copies, $0.25 each; 100 or more, $0.20
each.)

PUB TYPE

Information Analyses (070)

EDRS PRICEDESCRIPTORSMFO1 /PCO1 Plus Postage.

*College Faculty; *Data Collection; *Evaluation
Methods; Higher Education; Literature Reviews;
Reliability; Self Evaluation (Individuals); *Student
Evaluation of Teacher. Performance; *Teacher
Effectiveness; Validity

ABSTRACTThis paper attempts to summarize the conclusions of
the major reviews of the literature on student ratings of teaching.
It is an update of a paper by the same name published as IDEA Paper
No. 20 from the Center for Faculty Evaluation and Development in1988. Viewing student ratings as data rather than evaluations mayhelp to put them in proper perspective. Studies have considered themultidimensionality of student ratings and their reliability andvalidity. They have been compared to student learning outcomes, theself-ratings of the instructor, and the ratings of others, andpossible sources of bias have been studied. There are probably morestudies of student ratings than of all the other data used toevaluate college teaching combined, and there are certainly enoughstudies to allow some conclusions. In general, student ratings tend
to be statistically reliable, valid, and relatively free from bias
and need for control. Nevertheless, they are only one source of data
about teaching and must be used with multiple sources of data to makejudgments about all the components of teaching. (Contains 67
references.) (SLD)***********************************************************************

Reproductions supplied by EDRS are the best that can be madefrom the original document.
***********************************************************************

center for

scum(

EVALUATION &DEVELOP/11W

o iDE4 PAPERNo. 32

Division of Continuing Education
Kansas State University

September, 1995

Student Ratings of Teaching: The Research Revisited
U.S. DEPARTMENT OF EDUCATION

Office of Educational Research and ImprovementEDU

TIONAL RESOURCES INFORMATIONCENTER (ERIC)This document has been reproduced asreceived from the person or organization
originating it.
Minor changes have been made to
improve reproduction quality.
Points of view or opinions stated in thisdocument do not necessarily represent
official OERI position or policy.

PERMISSION TO REPRODUCE AND
DISSEMINATE THIS MATERIAL
HAS BEEN GRANTED BY

William E. Cashin
Kansas State University

Negative attitudes toward student ratings are especially resistant tochange, and it seems that faculty and administrators support theirbelief in student-rating myths with personal and anecdotal evidence,
which [for them] outweighs empirically based research evidence.
(Cohen, 1990, p. 124-125)

There are now more than 1,500 references dealing withresearch on student evaluations of teaching. IDEA
Paper No. 20, Student Ratings of Teaching: A Summary of the Research (Cashin, 1988) attempted tobriefly summarize the research from 1971 to 1988.
This paper is an update of that paper and repeats muchof its content. No major study published since then has
substantively changed that paper's conclusions, butseveral studies or reviews of the literature provide
modifications or further support for its conclusions.
This paper will attempt to summarize the conclusions of
the major reviews of the student rating literature from
Costin, Greenough, and Menges (1971) to the present.
That literature is extensive and complex. Obviously, apaper this brief can offer only broad, general conclusions and very limited citations. Interested readers areencouraged to consult the various reviews and theirindividual references for details. For readers with lesstime, both Braskamp and Ory (1994) and Centra (1993)
have chapters summarizing the student rating research; see also Davis (1993) and McKeachie (1994).
The ERIC descriptor for student ratings is "student
evaluation of teacher performance". I suggest that theterm "student ratings" is preferable to "student evaluations." "Evaluation" has a definitive and terminal
connotation; it suggests that we have an answer.
"Rating" implies that we have data which need to beinterpreted. Using the term "rating" rather than "evaluation" helps to distinguish between the people who
provide the information (sources of data) and thepeople who interpret it in combination with othersources of data (evaluators).

W/Lz/4A4 6.A5)YAJ
TO THE EDUCATIONAL RESOURCESINFORMATION CENTER (ERIC)

provides sufficient information to make a valid judgment
about overall teaching effectiveness. Further, there areimportant aspects of teaching that students are notcompetent to rate (see IDEA Paper No. 21, Defining and
Evaluating College Teaching, Cashin, 1989, for details.)

Multidimensionality
There have been a number of factor analytic studies(see Abrami & d'Apollonia, 1990; Feldman, 1976b;
Kulik & McKeachie, 1975; and Marsh & Dunkin, 1992,
for details) that conclude that student rating forms are
multidimensional, i.e., that they measure several
different aspects of teaching. Put another way, nosingle student rating item, nor set of related Items,
will be useful for all purposes.
Both Centra (1993) and Braskamp and Ory (1994)
identify six factors commonly found in student rating
forms:
1. Course organization and planning
2. Clarity, communication skills
3. Teacher student interaction, rapport
4. Course difficulty, workload
5. Grading and examinations
6. Student self-rated learning
Marsh's (1984) SEEQ (Students' Evaluations of Educational Quality) form has nine dimensions: learning/
value, enthusiasm, organization, group interaction,individual rapport, breadth of coverage, exams/grades,
assignments, and workload. Other student rating forms
have items measuring some or all of the above dimensions. In several of his reviews of the literature,
Feldman (1976b, 1983, 1984, 1987, and 1988) catego-

rized student ratings itemsand gave examplesinto
Viewing student ratings as data rather than as evaluations may also help to put them in proper perspective.
Writers on faculty evaluation are almost universal inrecommending the use of multiple sources of data. No

single source of dataincluding student rating data

as many as 22 different logical dimensions. In a more
recent review, Feldman (1989b) identified 28 dimensions. When interpreting student rating data, we must

distinguish among the various Items and theirdimensions to Insure that all of the appropriate©1995 Center for Faculty Evaluation and Development

2

BEST COPY AVAiLABLE

dimensions are rated. Averaging dissimilar items
is not appropriate.
Although there is general agreement that student
ratings are multidimensional, and that various dimensions should be used when their purpose is to improveteaching, there is disagreement about how many, or
which, dimensions should be used for personnel
decisions. In several articles Abrami (e.g., 1989a; andAbrami & d'Apollonia, 1991) suggested that one or a

few global or summary type items might provide
sufficient student rating data for personnel decisions. Centra (1993) and Braskamp and Ory (1994)
make a similar recommendation. Cashin and Downey(1992) tested this using the IDEA Overall Evaluation
measure as the criterion of teaching effectiveness.

Each of three global itemsindividuallyaccounted forat least 50% of the variance in the criterion measure:overall instructor effectiveness, 54%; overall course

worth, 60%; overall amount learned, 69%. However
contrary to their hypothesiscontrolling for the students' motivation to take the course, the size of theclass, or the difficulty of the subject matter, did not add
significantly to the amount of variance explained.
Marsh (1994) had some reservations about the way the
IDEA Overall Evaluation measure was calculated andhe generated four variations that he considered improvements. However, Cashin, Downey, and Sixbury(1994)using each of Marsh's four variations as thecriterion measureobtained the same results as theoriginal study: each of the global items accounted forat least 50% of the variance in each of Marsh's criterion
measures, and the control items added little.

Reliability
In the educational measurement literature, reliability
covers consistency, stability, and generalizability ofitems. For student rating items, reliability refers mostoften to consistency or interrater agreement (i.e.,
within a given class do the students tend to give similar

ratings on a given item). Reliability varies depending
upon the number of raters, i.e., the more raters, themore reliable. For example, with the IDEA system(Sixbury & Cashin, 1995a), the median reliabilities(intraclass correlations) for the 38 items are:for 10 raters, .69
for 15 raters, .83
for 20 raters, .83
for 30 raters, .88
for 40 raters, .91
Similar or higher reliabilities are typically found withother well-designed forms, i.e., forms developed withthe assistance of someone knowledgeable about
educational measurement. As a rule of thumb, I

recommend that items with fewer than ten raters(reliabilities below .70), be interpreted with particular caution.
Stability is concerned with agreement between raters
over time. In general, ratings of the same instructor
tend to be similar over time (Braskamp & Ory, 1994;
Centra, 1993). For example, a longitudinal study(Overall & Marsh, 1980) compared end-of-course

ratings with ratings by the same students years later (atleast one year after graduation). The average correlation was .83.

Generalizability is concerned with how confident wecan be that our data accurately reflect the instructor's
general teaching effectiveness, not just how effective
he or she was in that particular course that term. A
study conducted by Marsh (1982) illustrates the question. He studied data from 1,364 courses, dividingthem into four categories: the same instructor teaching
the same course but in different terms, the same
instructor teaching a different course, different
instructors teaching the same course, and different
instructors teaching different courses. This permitted
him to study the differential effects of the instructor andof the course. He then correlated student ratings in thefour different categories, separating items related to the
instructor (e.g., enthusiasm, organization, discussion)from background items (e.g., student's reason fortaking the course, workload). The average correlations
are shown below; the correlations in parentheses arefor the background items.
Same Course

Different Course

Same
Instructor

.71

.52

(.69)

(.34)

Different
Instructor

.14
(.49)

.06
(.21)

The instructor-related correlations were higher for thesame instructor, even when teaching a different course.
The correlations for the background items (in parenthe-ses)more tied to the course than the instructorwere
higher for the same course. Marsh concluded that the

instructor, not the course, is the primary determinant of the student rating items. Marsh's results arecomparable to other generalizability studies (Gillmore,
Kane, & Naccarato, 1978; and Hogan, 1973).
When making personnel decisions, we want to use thedata to make judgments about the instructor's generalteaching effectiveness. When considering student
ratings (remembering that we need other kinds ofinformation beyond student ratings), the following seemto be reasonable rules of thumb. If the instructor
teaches only one course (e.g., part-time instructors),consistent ratings from two different terms may besufficient. For most instructors, however, use ratingsfrom a variety of courses, for two or more courses
from every term for at least two years, totaling atleast five courses. If there are fewer than fifteen
raters in any of the classes, data from additionalclasses are recommended.

Validity
In educational measurement, the basic question
concerning validity is: does the test measure what it is
supposed to measure? For student ratings this trans-

lates into: to what extent do student rating items
measure some aspect of teaching effectiveness?
Unfortunately there is no agreed upon definition of

"effective teaching" nor any single, all-embracing
criterion. The best that one can do is to try various
approaches, collecting data that either support orcontest the conclusion that student ratings reflect
effective teaching.

from perfect, in part because many of the variables that
relate to students' learning will be related to student
characteristics (e.g., motivation or ability), not to
instructor characteristics.

Approach TwoInstructor's Self Ratings

Approach OneStudent Learning
Theoretically, the best criterion of effective teaching is
student learning. Other things being equal, the students of more effective teachers should learn more. A
number of studies have attempted to study this hypothesis by comparing multiple-section courses. In the
typical study, different instructors teach different
sections of the same course, using the same syllabus
and textbook, and most importantly using the same
external final exam, i.e., an exam developed by someone other than the instructors. Cohen (1981) and
Feldman (1989b) reviewed these studies. Using thestudents' grades on the external exam as the measureof student learning, they examined correlations between the exam grade and various student rating items.
The average correlations are given below (1981
Cohen; 1989Feldman):Researchers have sought for a criterion of effective
teaching that would be acceptable to faculty. One
possibility is the self ratings of the instructor. In areview of the literature, Feldman (1989a) cites 19
studies which correlated instructor's self ratings withstudent ratings. The average correlation was .29.
However, in one study (Marsh, Overall, & Kesler, 1979)
instructors were asked to rate two different courses inorder to see if the course the instructor rated higher
was also rated higher by the students. The mediancorrelationbased on six factor scores between the
instructor's self ratings and the students' ratingswas.49. In a later report (Marsh & Dunkin, 1992) using
nine factor scores, the median was .45. Such studies
provide further support for the validity of the students'
ratings.

Approach ThreeThe Ratings of Others
Student ratings ofachievement or learning
overall course
overall instructor
teacher skill dimension

1981

1989

.47
.47
.44
.50

.46

course preparation
clarity of objectives
teacher structure dimension

.57
.35

.47

understandablenessknowledge of subject
teacher rapport dimension

.56
.34
.31

availability
respect for students
teacher interaction dimension
encouraging discussion

.36
.23
.22
.36

Note on Interpreting Validity Correlations: Earlier I
suggested as a rule of thumb that reliability correlations of at least .70 (at least 10 raters) were desirable.
However, in the social sciences validity correlations
above .70 are unusual, especially if studying complex
phenomena, such as student learning. As a rule ofthumb, I suggest that student rating validity correlations
between .00 and .29, even when statistically significant,are not practically useful. Correlations between .30
and .49 are practically useful. Correlations between.50 and .70 are very useful but are not common whenstudying complex phenomena.
Using the above rule of thumb, the average correlations
reported by Cohen (1981) and Feldman (1989b) are
generally useful. These relationships tend to support
the validity of student ratings because the classes inwhich the students gave the instructor higher
ratings tended to be the classes where the students learned more, i.e., scored higher on the external exam. On the other hand, the correlations are far

If one is willing to grant that the ratings of administrators, colleagues, alumni, and others have some validityand, excepting alumni, that these ratings are
independent of feedback from studentsthen student
ratings share that validity.

Administrator's RatingsStudent ratings correlatewith administrator's ratings, ranging from .47 to .62(Kulik & McKeachie, 1975), but Feldman (1989a), using
global items, found a lower average correlation of .39.

Colleague's RatingsStudent ratings correlate withcolleague's ratings, .48 to .69 (Kulik & McKeachie,1975); Feldman (1989a) found an average of .55.
Marsh and Dunkin (1992) question the usefulness ofcolleague's ratings based on classroom visitation
because such ratings tend to be unreliable.
Some faculty question whether the students have anappropriate conception of what effective teaching is. Ina review of 31 studies, Feldman (1988) found that thestudents' view of effective teaching was very similar to
the faculty's view (average correlation equalled .71).
There were some differences in emphasis between thetwo groups. Students tended to place more weight on
the instructor being interesting, having good speaking
skills, and being available to help; students also focused more on the outcomes of instruction, e.g., what
they learned. Faculty placed relatively more weight on
intellectual challenge, motivating students, setting highstandards, and fostering student self-initiated learning.

Alumni RatingsStudent ratings correlate with alumni
ratings, .40 to .75 (Overall & Marsh, 1980; Braskamp &Ory, 1994). Feldman (1989a) found an averagecorrelation of .69. This belies the conventional wisdomthat the students will come to appreciate our teachingafter they get into the real world as working adults.

4

Trained ObserversA few studies have used external

the question of "fairness" in comparing instructors
teaching classes of interested students versus
instructors teaching classes of uninterested students.

observers who were trained (see Feldman, 1989a, also
Marsh & Dunkin, 1992). Reviewing five studies,
Feldman found positive correlations with global student
ratings (average correlation was .50). On a related
issue, in another study (Murray, 1983) the medianreliability for trained observers was .76. This suggeststhat peer ratings based on classroom observation
would be reliable if the observers were trained.

In the interest of clarity, rather than using "bias" in the
restricted sense I did in the original paper, I will identify
variables (when correlated with student ratings) that
require control, especially when making personnel
decisions.

Approach FourComparison with Student Com-

Variables Not Requiring Control

ments
Some faculty question the value of student ratings but
accept student written comments to open-ended
questions. One study (Ory, Braskamp, & Pieper, 1980)
of 14 classes found a correlation of .93 between a
global instructor item and the students comments. A
second study (Braskamp, Ory, & Pieper, 1981) of 60
classes found a correlation of .75. These studies
suggest that, for personnel decisions, the informationfrom student ratings overlaps considerably the information in student comments.

Approach Five Possible Sources of Bias

One need not talk with faculty very long to be aware of
their concern about possible biases in student ratings
about variables that correlate with student ratings.
Some writers have suggested that bias be defined asanything not under the control of the instructor. Marsh(1984) argued against this definition because, forexample, grading leniencyinstructors giving higher
grades than the students earnedwould not be considered a bias using this definition. Marsh suggests thatbias In student ratings should be restricted to vari-ables NOT related to teaching effectiveness. By thisdefinition, the correlations between student ratings andclass size, or the students' interest in the course are
not biases because it is probable that students in smallclasses, or classes of students who are interested in
the subject matter actually do learn more.

In IDEA Paper No. 20 (Cashin, 1988), I suggested aneven narrower definition when using ratings for personnel decisions or the instructor's improvement. I suggested restricting bias to variables not a function of the
instructor's teaching effectiveness. Thus, student
motivation or class size might impact teaching effectiveness, but instructors should not be faulted if they
were less effective teaching large classes of unmotivated students than their colleagues who were teachingsmall classes of motivated students. In this case,student motivation and class size, although related toteaching effectiveness, were not a function of the
instructor's characteristics, but of student and course
characteristics. Thus, they should be considered
sources of bias, and should be controlled for by using
appropriate comparative data. Feldman (1995, April)observedaccurately in my judgmentthat such a
definition of bias, while possibly acceptable, was not
the usual definition and it served to confuse the literature. Marsh and Dunkin (1992)considering that priorstudent interest in the subject matter is not a bias
because it does impact teaching and learningraise

Despite widespread faculty concern, the research has
uncovered relatively few variables that correlate withstudent ratings but are not related to instructional
effectiveness. Generally the following variables tend toshow little or no relationship to student ratings:

A. Instructor variables not related to student
ratings:
1) age, and teaching experiencein generalage, and also years of teaching experience, are notcorrelated with student ratings. However, where small
differences have been found, they tend to be negative,i.e., older faculty receive lower ratings (Feldman,1983). Marsh and Hocevar (1991) point out that mostof the studies have been cross-sectional, studying
different cohorts of faculty to represent different age
groups. In a longitudinal study they analyzed student
ratings of the same instructors for as long as 13 years.
They found no systematic changes over the years.
2) gender of the Instructorin a review of 14
laboratory or experimental studies, e.g., where students rated descriptions of fictitious teachers, Feldman(1992) found no differences in global ratings in the
majority of studies, but in a few studies the maleteachers received higher ratings. In a second review of
28 studies of actual ratings of real teachers reporting
global ratings, he (Feldman, 1993) found a very slightaverage difference in favor of women teachers (r=.02). However, a few studies raised the question ofwhether women faculty had to do more of what was
being rated (e.g., being available to students) to obtainthe same ratings as men. In a few other studies there
was a gender of student/gender of instructor interaction, i.e., female students rated female teachers higher,
and male students rated male instructors higher.
3) raceCentra (1993) points out that there havebeen hardly any studies of the race of the instructor.
He speculates that students of the same race as the
instructor might rate the instructor higher. In a doctoral
dissertation using IDEA, Li (1993) found no differencein the global ratings of Asian students compared toAmerican students of their (presumably Caucasian)
instructors.
4) personalityfew personality traits tend tocorrelate with student ratings (Braskamp & Ory, 1994;
Centra, 1993). In studies measuring personality using
instructor's self report (e.g., personality inventories,
self-description questionnaires), Feldman (1986) foundonly two (out of fourteen) traits that had averagecorrelation with a global item that approached practicalsignificant correlations. These traits were positive self

esteem (r= .30), and energy and enthusiasm (r =.27). Note, I suggest that these two traits enhance theinstructor's teaching effectiveness and so should not
be controlled. Murray, Rushton, and Paunonen (1990)
found significantly different patterns of personality traitsof psychology instructors teaching six different types ofcourses, e.g., introductory, graduate. They concluded
that instructors tend to be differentially suited to different types of courses.

5) research productivityhas little correlation
with student ratings (Centra, 1993). In his review of theliterature, Feldman (1987) found the average correlation between research productivity and overall teachingeffectiveness items to be .12. This very low correlationsuggests that research productivity is indicative neitherof good teaching nor bad teaching.

B. Student variables not related to student ratings:
1) age of the student(Centra, 1993).
2) gender of the student(Feldman, 1977,1993), but sometimes there is a gender ofstudent/gender of instructor interaction (seeabove under instructor variables).
3)

level of the studente.g., freshman

4)
5)

student's GPA(Feldman, 1976a).
student's personality(Abrami, Perry, &(McKeachie, 1979).
Leventhal, 1982).

C. Course variables not related to student ratings:
1) class sizealthough there is a tendency forsmaller classes to receive higher ratings, it is avery weak inverse association, i.e., smaller classes
receive higher ratings, average r = -.09 (Feldman,1984). The average correlation of class size forthe 38 IDEA items is -.14 (Sixbury & Cashin, 1995a).

of the content. The literature is complex (see Abrami,
Leventhal, & Perry, 1982), but Marsh and Ware (1982)
suggest that, especially in studies involving an incentive and a test, manipulations of instructor expressiveness primarily influences items related to instructor
enthusiasm, and manipulation of content coverage
primarily influences items related to instructor knowledge and student exam performance. Nevertheless,making the class interesting as well as informative
helps students learn content. Expressiveness tends toenhance learning and does NOT require control.

B. Student variables related to student ratings:
1) student motivationinstructors are more
likely to receive higher ratings in classes where students had a prior interest in the subject matter (Marsh& Dunkin, 1992), or were taking the course as anelective (Aleamoni, 1981; Braskamp & Ory, 1994;
Centra, 1993; Feldman, 1978). The average correlation of the IDEA (Sixbury & Cashin, 1995a) motivation
item, "I had a strong desire to take this course," withthe other 37 items is .40. Marsh and Dunkin (1992)
conclude that reason for taking the course (which
overlaps with student motivation), also is related to
student ratings. Higher ratings were received fromstudents who took a course for general interest, or as amajor elective; lower ratings were received when thecourse is being taken as a major requirement or a
general education requirement. This variable REQUIRES CONTROL.

2) expected gradesthere tend to be positive,2) time of day when the course is taught

but low correlations (.10 to .30) between students
ratings and expected grades (Braskamp & Ory, 1994;
Feldman, 1976a; Howard & Maxwell, 1980 and 1982;
Marsh & Dunkin, 1992). Three possible hypotheses
have been proposed for these correlations. One is the(Aleamoni, 1981; Feldman, 1978).

validity hypothesisthe students who learned more

D. Administrative variables not related to student
ratings:
1) time during the term when ratings are collected; any time during the second half seems
to yield similar ratings(Feldman, 1979).

Variables Possibly Requiring Control
The research cited above suggests that many
variables suspected of biasing student ratings are notcorrelated with them to any practically significant
degree. For the following variables, however, theresearch suggests that there are correlationsrelationshipswith student ratings that may require control.

A. Instructor variables related to student ratings:
1) faculty rankregular faculty tend to receive
higher ratings than graduate teaching assistants(Braskamp & Ory, 1994). This variable does NOT
require control because regular faculty as a group tend
to be more effective teachers than GTAs as a group.

2) expressivenessthe Dr. Fox effect (Naftulin,
Ware, & Donnelly, 1973)where a professional actor
delivering little content received high ratingssuggeststhat student ratings may be more influenced by an
instructor's style of presentation than by the substanceearn higher grades and give higher ratings (therefore,
student ratings are valid). Another explanation is

grading leniencyinstructors giving higher grades
than the students deserve receive higher ratings thanthey deserve. A third is based on student character-isticssome student characteristics, e.g., high motivation, lead to greater learning and, therefore, to higher
grades and higher ratings. In two studies by Howardand Maxwell (1980 & 1982), which used IDEA data,
they concluded that most of the correlation between
expected grade and a global instructor item wasaccounted for by student (self-reported) learningthe

validity hypothesisand desire to take the coursea
student characteristic. To control for the possibility ofgrade leniency, my recommendation is to have peers(faculty knowledgeable in the subject matter) review thecourse material, particularly exams, computer scoredtest results, graded samples of essays, projects, etc.;
and judge whether grades are inflated.

C. Course variables related to student ratings:
1) level of the coursehigher level courses,
especially graduate courses, tend to receive higher
ratings (Aleamoni, 1981; Braskamp & Ory, 1994;
Feldman, 1978). However, the differences tend to besmall. Regarding possible control, check to see if your6

freshman/sophomore classes receive lower ratings
than your junior/senior classes; similarly compare
undergraduate with graduate classes. If yes, do thedifferences remain after controlling for student motivation and size? If yes, develop comparative data for theappropriate levels.

2) academic fieldFeldman (1978) reviewed
some studies showing that humanities and arts typecourses receive higher ratings than social science typecourses, which in turn receive higher ratings than mathscience type courses. Others (Braskamp & Ory, 1994;
Cashin, 1990; Centra, 1993; Marsh & Dunkin, 1992;
and Sixbury & Cashin, 1995b) have found similar
results. Although there is increasing evidence thatratings for different fields differ, it is not clear why.
Cashin (1990) suggests six possible explanations. For
example, if some fields are rated lower because they
are more poorly taught, then these differences do not
require control. On the other hand, if instructors infields requiring more quantitative reasoning skills are
rated lower because today's students are less competent in such skillsone of the hypotheses explaining
why some fields are rated lowerthen this should becontrolled for.

3) workload/difficultythese are correlated withstudent ratings (Centra, 1993; Marsh & Dunkin, 1992).
However, contrary to faculty belief, they are correlated
positively, i.e., students give higher ratings in difficult
courses where they have to work hard. Although
positive, the correlations are not large. For example,
using the 38 IDEA items (Sixbury & Cashin, 1995a) the
average correlations with the remaining 37 IDEA items
are:Amount of reading.11
Amount of other (non reading) assignments.16
Difficulty of subject matter.15
Worked harder in this course.29
These modest results support the validity of student
ratings and the variables do NOT require control.

D. Administrative variables related to student
ratings:
1) non-anonymous ratingssigned ratings tend
to be higher (Braskamp & Ory, 1994; Centra, 1993;
Feldman, 1979; Marsh & Dunkin, 1992). The hypothesis is that requiring students to sign their names
inflates the ratings because some students are concerned about possible reprisals. Control: instruct thestudents not to sign their ratings.

2) Instructor present while students complete
ratingsthese tend to be higher (Braskamp & Ory,
1994; Centra, 1993; Feldman, 1979; Marsh & Dunkin,1992), possibly for the same reason as non-anonymous ratings. Control: have the instructor leave theroom while the ratings are being completed and collected.

3) purpose of the ratingssome studies havefound that if the directions say the ratings will be used
for personnel decisions, the ratings tend to be higher
than if they will be used only by the instructor for
improvement (Braskamp & Ory, 1994; Centra, 1993;
Feldman, 1979; Marsh & Dunkin, 1992). Speculation isthat the students tend to be lenient if the data will be

used by someone other than the instructor. Control:
include in the standard directions the purpose(s) for
which the ratings will be used. This will not eliminate
the bias, but it will eliminate variations in ratings due todifferences in student beliefs about their purpose.

Usefulness of Student Ratings
Many faculty will grant the usefulness of student ratings
for personnel decisions, but question their usefulness
for improvement, preferring to rely on students' openended comments. Cohen (1980) performed a metaanalysis of 17 studies of the effect of student-rating
feedback on improving teaching. Receiving feedback
about student ratings administered during the first halfof the term was positively related to improving college
teaching as measured by student ratings administeredat the end of the term. Typically there were threegroups. All groups had ratings administered during thefirst half of the semester and again at the end. That is
all the first group received, i.e., no feedback. The
second group received the student rating feedback,quantitative data, from the first student ratings. In
addition to that, the third group received some kind of
consultation (which varied across the different studies).
Using the end-of-term ratings as the measure of
improvement and setting the first group's mean ratings
at the 50th percentile, Cohen presented the following
data:
During term

No student rating feedback =Only student rating feedback =Student rating feedback plus
consultation =End of Term

50th %ile
58th %Ile
74th %ite

Conclusion, if an institution really intends to use
student ratings to improve teaching, it needs to provide
some kind of consultation to the instructors.

Conclusion
There are probably more studies of student ratings
than of all of the other data used to evaluate college
teaching combined. Although one can find individual
studies that support almost any conclusion, for anumber of variables there are enough studies todiscern trends. In general, student ratings tend to bestatistically reliable, valid, and relatively free from bias
or the need for control; probably more so than any
other data used for evaluation. Nevertheless, student
ratings are only one source of data about teaching andmust be used in combination with multiple sources ofdata if one wishes to make a judgment about all of thecomponents of college teaching. Further, student
ratings are data that must be interpreted. We should
not confuse a source of data with the evaluators who
use student rating datain combination with otherkinds of datato make their judgments about an
instructor's teaching effectiveness.

References and Suggested Reading
Abrami, P. C. (1989a). How should we use student ratings toevaluate teaching? Research in Higher Education, 30, 221-227.

Cohen, P. A. (1980). Effectiveness of student-rating feedback for
improving college instruction: A meta-analysis of findings.
Research in Higher Education, 13, 321-341.

Abrami, P. C. (1989b). SEEQing the truth about student ratings of
instruction. Educational Researcher, 43, 43-45.

Cohen, P. A. (1981). Student ratings of instruction and student
achievement: A meta-analysis of multisection validity studies.
Review of Educational Research, 51, 281-309.

Abrami, P. C., & d'Apollonia. (1990). The dimensionality of ratings
and their use in personnel decisions. In M. Theall, & J. Franklin(Eds.), Student ratings of instruction: Issues for improving
practice: New Directions for Teaching and Learning, No. 43 (pp.
97-111). San Francisco: Jossey-Bass.

Cohen, P. A. (1990). Bring research into practice. In M. Theall, & J.
Franklin (Eds.), Student ratings of instruction: Issues for
improving practice: New Directions for Teaching and Learning,
No. 43 (pp. 123-132). San Francisco: Jossey-Bass.

Abrami, P. C., & d'Apollonia. (1991). Multidimensional students'
evaluations of teaching effectiveness- generalizability of "N = 1"
research: Comments on Marsh (1991). Journal of Educational
Psychology, 83, 411-415.

Costin, F., Greenough, W. T., & Menges, R. J. (1971). Student
ratings of college teaching: Reliability, validity, and usefulness.
Review of Educational Research, 8, 511-535.

Davis, B. G. (1993). Tools of teaching. San Francisco: JosseyBass.

Abrami, P. C., Leventhal, L., & Perry, R. P. (1982). Educational
seduction. Review of Educational Research, 52, 446-464.
Abrami, P. C., Perry, R. P., & Leventhal, L. (1982). The relationshipbetween student personality characteristics, teacher ratings, andstudent achievement. Journal of Educational Psychology, 74,
111-125.

Feldman, K. A. (1976a). Grades and college students' evaluationsof their courses and teachers. Research in Higher Education, 4,
69-111.
Feldman, K. A. (1976b). The superior college teacher from thestudents' view. Research in Higher Education, 5, 243-288.

Aleamoni, L. M. (1981). Student ratings of instruction. In J. Millman(Ed.), Handbook of teacher evaluation (pp. 110-145). Beverly
Hills, CA: Sage.

Feldman, K. A. (1977). Consistency and variability among college
students in rating their teachers and courses: A review andanalysis. Research in Higher Education, 6, 233-274.

Benton, S. E. (1982). Rating college teachers: Criterion validity
studies of student evaluation-of-instruction instruments. AAHEERIC Higher Education Research Report, No. 1. Washington,
DC: American Association for Higher Education.

Feldman, K. A. (1978). Course characteristics and college students'
ratings of their teachers; what we know and what we don't.
Research in Higher Education, 9, 199-242.

Braskamp, L. A., & Ory, J. C. (1994). Assessing faculty work:
Enhancing individual and institutional performance. San
Francisco: Jossey-Bass.
Braskamp, L. A., Ory, J. C., & Pieper, D. M. (1981). Student writtencomments: Dimensions of instructional quality. Journal of
Educational Psychology, 73, 65-70.

Feldman, K. A. (1979). The significance of circumstances forcollege students' ratings of their teachers and courses. Researchin Higher Education, 10, 149-172.
Feldman, K. A. (1983). Seniority and experience of college teachers
as related to evaluations they receive from students. Research inHigher Education, 18, 3-124.
Feldman, K. A. (1984). Class size and college students' evaluationsof teachers and courses: A closer look. Research in Higher
Education, 21, 45-116.

Cashin, W. E. (1988). Student ratings of teaching: A summary of
the research. IDEA Paper No. 20. Manhattan, KS: Kansas State
University, Center for Faculty Evaluation and Development.
(ERIC Document Reproduction Service No. ED 302 567)
(Reprinted in K. A. Feldman & M. B. Paulsen (Eds.). (1994).
Teaching and learning in the college classroom (pp. 531-541).
Needham Heights, MA: Ginn Press).

Feldman, K. A. (1986). The perceived instructional effectiveness ofcollege teachers as related to their personality and attitudinal
characteristics: A review and synthesis. Research in Higher
Education, 24, 129-213.

Cashin, W. E. (1989). Defining and evaluating college teaching.
IDEA Paper No. 21. Manhattan, KS: Kansas State University,
Center for Faculty Evaluation and Development. (ERIC Docu:ment Reproduction Service No. ED 339 731)

Feldman, K. A. (1987). Research productivity and scholarly
accomplishment of college teachers as related to their instructional effectiveness: A review and exploration. Research inHigher Education, 26, 227-298.

Cashin, W. E. (1990). Students do rate different academic fields
differently. In M. Theall, & J. Franklin (Eds.), Student ratings of
instruction: Issues for improving practice: New Directions for
Teaching and Learning, No. 43 (pp. 113-121). San Francisco:
Jossey-Bass.

Feldman, K. A. (1988). Effective college teaching from the students'and faculty's view: Matched or mismatched priorities. Researchin Higher Education, 28, 291-344.

Cashin, W. E. (1992). Student ratings: The need for comparative
data. Instructional Evaluation and Faculty Development, 12, 1-6.
Cashin, W. E., & Downey, R. G. (1992). Using global student
ratings for summative evaluation. Journal of Educational
Psychology, 84, 563-572.
Cashin, W. E., Downey, R. G., & Sixbury, G. R. (1994). Global andspecific ratings of teaching effectiveness and their relation tocourse objectives: Reply to Marsh (1994). Journal of Educational Psychology, 86, 649-657.

Centre, J. A. (1993). Reflective faculty evaluation: Enhancing
teaching and determining faculty effectiveness. San Francisco:
Jossey-Bass.

Feldman, K. A. (1989a). Instructional effectiveness of college
teachers as judged by teachers themselves, current and former
students, colleagues, administrators and external (neutral)observers. Research in Higher Education, 30, 137-194.
Feldman, K. A. (1989b). The association between student ratings ofspecific instructional dimensions and student achievement:
Refining and extending the synthesis of data from multisection
validity studies. Research in Higher Education, 30, 583-645.
Feldman, K. A. (1992). College students' views of male and female
college teachers: Part I-Evidence from the social laboratory andexperiments. Research in Higher Education, 33, 317-375.

Feldman, K. A. (1993). College students' views of male and female
college teachers: Part II-Evidence from students' evaluations of
their classroom teachers. Research in Higher Education, 34,
151-211.

Feldman, K. A. (1995, April). Some unresolved issues in studying
instructional effectiveness and student ratings. Invited address
presented at the annual meeting of the American Educational
Research Association, San Francisco, CA.
Gillmore, G. M., Kane, M. T., & Naccarato, R. W. (1978). The
generalizability of student ratings of instruction: Estimation of theteacher and course components. Journal of Educational
Measurement, 15, 1-13.

Hogan, T. P. (1973). Similarity of student ratings across instructors,
courses, and time. Research in Higher Education, 1, 149-154.

Marsh, H. W., Overall, J. U., & Kesler, S. P. (1979). Validity ofstudent evaluations of instructional effectiveness: A comparisonof faculty self-evaluations and evaluation by their students.
Journal of Educational Psychology, 71, 149-160.

Marsh, H. W., & Ware, J. E. (1982). Effects of expressiveness,
content coverage, and incentive on multidimensional student
rating scales: New interpretations of the Dr. Fox effect. Journalof Educational Psychology, 74,126-134.

McKeachie, W. J. (1979). Student ratings of faculty: A reprise.
Academe, 65,384-397.
McKeachie, W. J. (1990). Research on college teaching: The
historical background. Journal of Educational Psychology, 82,
189-200.

Howard, G. S., & Maxwell, S. E. (1980). The correlation between
student satisfaction and grades: A case of mistaken causation?
Journal of Educational Psychology, 72, 810-820.

McKeachie, W. J. (1994). Teaching tips: Strategies, research, andtheory for college and university teachers. (9th ed.). Lexington,
MA: D. C. Heath.

Howard, G. S., & Maxwell, S. E. (1982). Do grades contaminate
student evaluations of instruction? Research in Higher Education, 16, 175-188.

Murray, H. G. (1980). Evaluating university teaching: A review of
research. Toronto, Canada: Ontario Confederation of University
Faculty Association.

Kulik, J. A., & McKeachie, W. J. (1975). The evaluation of teachers
in higher education. In F. N. Kerlinger (Ed.), Review of researchin education (Vol. 3, pp. 210-240). Itasca, IL: F. E. Peacock.

Murray, H. G. (1983). Low-inference classroom teaching behaviors
and student ratings of college teaching effectiveness. Journal of
Educational Psychology, 75, 138-149.

Li, Y. (1993). A comparative study of Asian and American students'
perceptions of faculty teaching effectiveness at Ohio University.
Unpublished doctoral dissertation, Ohio University, Athens.

Murray, H. G., Rushton, J. P., & Paunonen, S. V. (1990). Teacher
personality traits and student instructional ratings in six types ofuniversity courses. Journal of Educational Psychology, 82, 250-

Marsh, H. W. (1982). The use of path analysis to estimate teacher
and course effects in student rating's of instructional effectiveness. Applied Psychological Measurement, 6,47-59.

Naftulin, D. H., Ware, J. E., & Donnelly, F. A. (1973). The Doctor
Fox lecture: A paradigm of educational seduction. Journal of
Medical Education, 48, 630-635.

Marsh, H. W. (1984). Students' evaluations of university teaching:
Dimensionality, reliability, validity, potential biases, and utility.
Journal of Educational Psychology, 76, 707-754.

Ory, J. C., Braskamp, L. A., & Pieper, D. M. (1980). Congruency ofstudent evaluative information collected by three methods.
Journal of Educational Psychology, 72, 181-185.

Marsh, H. W. (1987). Students' evaluations of university teaching:
Research findings, methodological issues, and directions for
future research. International Journal of Educational Research,
11, 253-388.

Overall, J. U., & Marsh, H. W. (1980). Students' evaluations of
instruction: A longitudinal study of their stability. Journal of
Educational Psychology, 72, 321-325.

Marsh, H. W. (1991a). Multidimensional students' evaluations ofteaching effectiveness: A test of alternative higher-order
structures. Journal of Educational Psychology, 83,285-296.
Marsh, H. W. (1991b). A multidimensional perspective on students'
evaluations of teaching effectiveness: Reply to Abrami andd'Apollonia (1991). Journal of Educational Psychology, 83, 416421.

Marsh, H. W. (1994). Weighting for the right criteria in the IDEA
System: Global and specific ratings of teaching effectiveness andtheir relation to course objectives. Journal of Educational
Psychology, 86, 631-648.
Marsh, H. W., & Dunkin, M. (1992). Students' evaluations ofuniversity teaching: A multidimensional perspective. In J. C.
Smart (Ed.) Higher education: Handbook of theory and research(Vol. 8, pp. 143-233). New York: Agathon.
Marsh, H. W., & Hocevar, D. (1991). Students' evaluations ofteaching effectiveness: The stability of mean ratings of the sameteachers over a 13-year period. Teaching & Teacher Education,
7,303-314.
Marsh, H. W., & Overall, J. U. (1979). Long-term stability ofstudents' evaluations: A note on Feldman's consistency andvariability among college students in rating their teachers andcourses. Research in Higher Education, 10, 139-147.

261.

Perry, R. P. (Ed.). (1990). Special Section: Instruction in higher
education. Journal of Educational Psychology, 82, 183-274.
Sixbury, G. R., & Cashin, W. E. (1995a). IDEA technical report no.
9: Description of database for the IDEA Diagnostic Form.
Manhattan: Kansas State University, Center for Faculty Evaluation and Development.
Sixbury, G. R., & Cashin, W. E. (1995b). IDEA technical report no.
10: Comparative data by academic field. Manhattan: Kansas
State University, Center for Faculty Evaluation and Development.
Theall, M., & Franklin, J. (Eds.). (1990). Student ratings of instruction: Issues for improving practice: New Directions for Teachingand Learning, No. 43. San Francisco: Jossey-Bass.

IDEA Papers may be ordered from the Center. Individual copies are$2.00. A complete set of IDEA Papers may be ordered for $15.00.
Bulk orders of the same paper: 20-49 copies are 30 cents a copy,
50-99 copies are 25 cents a copy, 100 or more copies are 20 centsa copy. Orders of less than $50.00 must be prepaid. Prices
effective through 6/30/96.
Center for Faculty Evaluation and Development
Kansas State University
1615 Anderson Avenue
Manhattan, KS 66502-4073

ADDENDUM-IDEA PAPER NO. 32

Add the following as the last paragraph of the paper.
This paper has summarized the general conclusions from the research on
student ratings. Whether those conclusions hold true for any given campus is anempirical question. If an institution has reason to believe that they do not apply, it
should gather local data to answer the question. However, in the absence ofevidence to the contrary, I suggest that the general conclusions serve as a guide.
10/95

WEC

off-90?

ON

ERIC

U.S. Department of Education
Office of Educational Research and Improvement (OERI)
Educational Resources Information Center (ERIC)

1

REPRODUCTION RELEASE(Specific Document)

I. DOCUMENT IDENTIFICATION:
Title:The Research Revisited, IDEA Paper No. 32

Student Ratings of Teaching:

Author(s):William E. Cashin

Corporate Source:
Kansas State University Center for Faculty Evaluation andDevelopment

IPublication Date:s eptember, 1995
I

II. REPRODUCTION RELEASE:
In order to disseminate as widely as possible timely and significant materials of interest to the educational community, documents announced
in the monthly abstract journal of the ERIC system, Resources In Education (RIE), are usually made available to users in microfiche, reproduced
paper copy, and electronic/optical media, and sold through the ERIC Document Reproduction Service (EDRS) or other ERIC vendors. Credit isgiven to the source of each document, and, if reproduction release is granted, one of the following notices is affixed to the document.

If permission Is granted to reproduce and disseminate the identified document, please CHECK ONE of the following two options and sign atthe bottom of the page.
The sample sticker shown below will beaffixed to all Level 1 documents

The sample sticker shown below will beaffixed to all Level 2 documents

PERMISSION TO REPRODUCE AND
DISSEMINATE THIS MATERIAL
HAS BEEN GRANTED BYPERMISSION TO REPRODUCE AND
DISSEMINATE THIS
MATERIAL IN OTHER THAN PAPER
COPY HAS BEEN GRANTED BY

I

Check here\e,

For Level 1 Release:
Permitting reproduction in
microfiche (4' x 6" film) orother ERIC archival media(e.g., electronic or optical)
and paper copy.

LI

I

Check here
For Level 2 Release:
Permitting reproduction in

co?fr

TO THE EDUCATIONAL RESOURCESINFORMATION CENTER (ERIC)TO THE EDUCATIONAL RESOURCESINFORMATION CENTER (ERIC)

microfiche (4' x 6' film) orother ERIC archival media(e.g., electronic or optical),but not in paper copy.

Level 2

Level 1Documents will be processed as Indicated provided reproduction quality permits. If permission
to reproduce is granted, but neither box is checked, documents will be processed at Level 1.

hereby grant to the Educational Resources Information Center (ERIC) nonexclusive permission to reproduce and disseminate
this document as indicated above. Reproduction from the ERIC microfiche or electronic./optical media by persons other than
ERIC employees and its system contractors requires permission from the copyright holder. Exception is made for non-profit
reproduction by libraries and other service agencies to satisfy information needs of educators in response to discrete inquiries.*

Sign

Printed Name/Position/Tide:

Signature:

here-)
pleaseWilliam E. Cashin, Director
Organization/Address:!Tone:
eleph

inkx

:

(913) 532-5637(913) 532-5970
Center for Faculty Evaluation and DevelDevelopment
Kansas State University
i bate:I Elvtatl Address:
5/24/96
1615 Anderson Ave.
idea@ksu.ksu.edu
Manhattan, KS 66502-4073
1I.

III. DOCUMENT AVAILABILITY INFORMATION (FROM NON-ERIC SOURCE):
If permission to reproduce is not granted to ERIC, or, If you wish ERIC to cite the availability of theplease provide the following information regarding the availability of the document. (ERIC will not document from another source,
announce a document unless it ispublicly available, and a dependable source can be specified.
Contributors should also be aware that ERIC selection criteria aresignificantly more stringent for documents that cannot be made available through EDRS.)
Publisher/Distributor:

Address:

Price:

IV. REFERRAL OF ERIC TO COPYRIGHT/REPRODUCTION RIGHTS
If the right to grant reproduction release is held by someone other than the

HOLDER:

addressee, please provide the appropriate name and address:

Name:

Address:

V. WHERE TO SEND THIS FORM:
Send this form to the following ERIC Clearinghouse:

Acquisitions
ERIC Clearinghouse
George Washington University
One Dupont Circle, NW, Suite 630
Washington, DC 20036-1183
However, if solicited by the ERIC Facility, or if making an unsolicitedcontribution to ERIC, return this form (and the document being
contributed) to:

ERIC Processing and Reference Facility
1301 Piccard Drive, Suite 100
Rockville, Maryland 20850-4305
Telephone: 301-258-5500
FAX: 301-948-3695
Toll Free: 800-799-3742

e-mail: erIcfac@ineted.gov(Rev. 3/96/96)