Multidimensional Students' Evaluations of Teaching Effectiveness: A Profile Analysis.

The literature on students' evaluations of teaching effectiveness (SETE) consists of thousands of studies and dates back to the 1920s and earlier |13~. In reviews of this research, Marsh |10, 12, 13, 16~ concluded that SETEs are: (a) multidimensional; (b) reliable and stable; (c) primarily a function of the instructor who teaches a course rather than of the course that is taught; (d) relatively valid against a variety of indicators of effective teaching; (e) relatively unaffected by a variety of variables hypothesized as potential biases to the ratings; and (f) seen to be useful by faculty as feedback about their teaching, by students for use in course selection, by administrators for use in personnel decisions, and by researchers. Based on his review, Marsh argued that SETEs are probably "the most thoroughly studied of all forms of personnel evaluation, and one of the best in terms of being supported by empirical research" |13, p. 369~. SETEs, like the teaching that they represent, are a multidimensional construct (for example, an instructor may be organized but lack enthusiasm). This contention is supported by common sense and a considerable body of empirical research |13~. The SETE literature contains several examples of well-constructed instruments with clearly defined factor structures that provide measures of distinct SETE scales. In his review, Marsh |13~ noted that the systematic approach used in the development of these instruments and the similarity in the factors identified by each supports their construct validity. The Students' Evaluation of Educational Quality instrument that is the focus of the present investigation appears to measure the most broadly representative set of scales and to have the strongest factor analytic support of these instruments |10, 11, 12, 13~. Marsh and Hocevar |17~ factor analyzed SEEQ responses for a total group of 24,158 sets of class average ratings and for each of 21 different subgroups selected to differ in terms of instructor rank, course level (undergraduate and graduate), and academic discipline. The set of 9 factors that SEEQ is designed to measure was identified in all 22 factor analyses, and factor scores based on the total group were highly correlated with factor scores based on separate analyses of each of the 21 subgroups (mean r |is greater than~ 0.99). The results demonstrated that SEEQ measures a broadly representative set of evaluation factors and supported the generalizability of the SEEQ factor structure. A logical extension of the multidimensionality of SETEs is that a given instructor has a distinct profile of SETE ratings (for example, high in organization but low in enthusiasm) that generalizes over time and across different courses. There is apparently no research that has examined this hypothesis, and so the purpose of this investigation is to test it. Support for existence of a distinguishable profile that is specific to each instructor has important implications for understanding SETEs, for the use of SETEs as feedback, and for the relation of SETEs to other criteria, such as student learning. The necessary starting point of such research is to determine whether instructors have distinguishable SETE profiles, and this is the purpose of the present investigation. Generalizability of SETEs Studies of the generalizability of SETEs have typically considered global ratings, total scores, or separate analyses of specific SETE dimensions. Researchers |for example, 4, 6, 8, 9~ have examined correlations between ratings of the same instructor in different offerings of the same course, the same instructor in different courses, and different instructors teaching the same course in an attempt to disentangle the relative influence of the course and the instructor. This research examined the generality of SETEs and the relative importance of the instructor who teaches the course and the course that is being taught. For example, for overall instructor and course ratings, the correlations between ratings of different instructors teaching the same course (one estimate of the course effect) were -0. …