Assessing Teaching Effectiveness in a Basic Foods Laboratory Setting: Phase Four - Pilot Testing the Instrument

This is the fourth phase of a research project to develop a student evaluation tool (SET) to evaluate teaching effectiveness in a basic foods laboratory taught at senior institutions. The goal of phase four is to test the instrument among students enrolled in basic foods laboratories. Four hundred seventy five students at eight U.S. universities who were enrolled in a basic food laboratory completed the SET and a demographic questionnaire. Reliability using Cronbach’s coefficient alpha was . 962 and factor analysis yielded 3 factors comparable to those identified by the Delphi Panel. Results from the first part of pilot testing indicate a reliable instrument preferred by students enrolled in basic food laboratory classes. INTRODUCTION To ensure continuous improvement, providing and receiving performance feedback is a key concept in any job. Within the academy, educators are routinely evaluated by students (and their peers) using some form of evaluation tool. An evaluation tool serves many functions: (1) providing feedback, (2) identifying training and development needs, (3) making decisions linking rewards and discipline to performance, (4) evaluating policies, programs, or training, and (5) validating the selection process (Woods, 2002). Within the academic community, all five of these functions are applicable to evaluating classroom teaching. Most educators (including the authors of this research) would contend that different evaluation tools should be implemented in the various types of classroom environments. A traditional, lecture style class is different than a laboratory setting. It follows, then, that these two disparate types of learning environments should not use the same teaching effectiveness evaluation tool. Frequently the same evaluation form is used, with the exception that the word “classroom” is replaced with “laboratory. ” The most critical question is: does this simple change allow for a truly accurate and fair evaluation of instruction in the laboratory classroom setting? Purpose of the Study The research and findings described in this manuscript represent the fourth phase in two projects that began in 2003. The original project developed a four phase process and SET to evaluate teaching in quantity foods laboratory classes (Chandler, Weber, and Finley, 2007). The current project applies the four phase process to developing a SET to evaluate teaching in a basic food laboratory. In phase one of the current project, a Delphi panel was organized and through three rounds, developed a SET to evaluate teaching in a basic food laboratory (Chandler, Weber, Finley, and Evans, 2008). Phase two of the 1 Weber et al.: Revisions to MS #1071 Published by ScholarWorks@UMass Amherst, 2009 development process consisted of surveying members of the Council for Hotel, Restaurant, and Institutional educators (CHRIE) about the appropriateness of each of the 27 criteria in evaluating teaching effectiveness. Phase three consisted of student evaluation of the preliminary SET. Based on recommendations from faculty and students, a SET consisting of 27 criteria was developed (Weber, Chandler, and Finley, submitted for publication). Phase four is a pilot study of the SET with students enrolled in a basic food laboratory class and is the focus of this study. REVIEW OF LITERATURE The literature review covered several subjects pertinent to the development of a student evaluation tool to be used in basic food laboratories. This particular type of laboratory class varies greatly from any standard university lecture course. Current evaluation tools such as the ones required by accrediting agencies are designed for the evaluation of standard lecture classes and are considered unsuitable for evaluation of basic food laboratory courses. In addition, it appears that there have been no studies aimed at collecting student evaluations of teaching effectiveness in basic food laboratory courses. Student Evaluations of Teaching University classes are commonly rated using different methods to ensure teaching effectiveness and course content. The most commonly used measure is the student evaluation tool (SET). These SETs are intended to provide valuable feedback to teachers and administrators. They are administered at the end of a course and evaluate only the student perceptions of effective teaching. According to Clayson (1999), SETs are used at four out of five campuses nationwide. There are questions, however, as to whether the results from SETs alone give a one-dimensional limited view of the classroom. The SET is viewed as one-dimensional because it does not include the perspective of other educators (Marsh and Roche, 1997). Research on developing SETs for hospitality management lecture courses has been reported by Gursoy and Umbreit (2005). The authors of this study of over 25,000 students proposed that student perceptions of the teacher’s organization, workload, and instructional abilities influence their perception of learning. The model explained 78% of the variance in the perception of learning. A study by Simpson and Siguaw (2000) examined faculty perceptions of student evaluations of teaching (SET) and their problems, to ascertain faculty response to SETs and perceptions to the responses, and to consider solutions to perceived problems with SETs. The researchers used a questionnaire to collect the data from a sample of 52 respondents from 15 different states in the United States and other locations outside of the United States. The faculty agreed that there are a number of problems with SETs including the fact that some students may use the SETs as a way of revenge, the question of validity of the evaluation instrument, and the ability of the student to objectively evaluate faculty (Simpson and Siguaw, 2000). McKeachie (1997), d’Apollonia and Abrami (1997), and Greenwald and Gillmore (1997), all conducted research into the validity of student ratings. These authors stated that faculty use ratings as feedback for how to improve their courses. Their conclusion claimed that student ratings are the most valid data source on teaching effectiveness. However, Marsh and Roche (1997) also stated that multidimensionality in teaching is important not only because of its obvious diagnostic utility as instructor feedback, but also because it provides a more sophisticated and realistic assessment of the various aspects of teaching. This review shows the strengths of using student ratings as a valuable part of improving teaching effectiveness. It must be recognized that it is the choice of the instructor to incorporate this feedback. Likewise, the review also discusses challenges in using a SET because students can have a limited (one-dimensional) view of the classroom/laboratory. In addition, students may not have enough experience in evaluation techniques to provide quality feedback. The researchers found no studies that attempted to compare educator and student feedback on criteria evaluating teaching effectiveness in foods laboratory courses. METHODOLOGY In September 2008, the Principal Investigator established contact with the faculty of several of the leading hospitality management programs to solicit their participation in this phase of the research. Ten universities agreed to participate in the project during the Fall 2008 semester and the Spring 2009 semester. In November 2008, appropriate quantities of the basic foods laboratory student evaluation tool (SET) developed during the first 3 phases of the project were distributed to each of the participating institutions, along with scantron sheets to facilitate data 2 International CHRIE Conference-Refereed Track, Event 8 [2009] http://scholarworks. mass.edu/refereed/Sessions/Wednesday/8 entry. The evaluation instrument was modified to collect demographic information from the responding students. Faculty at the participating institutions were asked to administer the evaluation instrument before the end of November, and they were provided with self-addressed, postage-paid envelopes for returning the completed instruments to the researchers. Data were received for the Fall 2008 semester from eight universities, and arrangements are in place to replicate this procedure during the Spring 2009 semester. The survey instrument consisted of the 27 criteria which asked students to rate their instructor on the items according to a 5-point Likert scale that ranged from “never to always,” or from “unacceptable to outstanding,” according to the wording of the criteria, and with 5 being the highest rating and 1 being the lowest rating. The evaluation instrument is available from the authors. Demographic information requested included students’ gender, class level, career aspiration, and grade they expected to earn in the course. Respondents were also asked to compare this evaluation instrument with the one currently being used at their university. Data analyses were conducted using SPSS (v. 15.0). The first stage of the analysis explored and compared descriptive statistics for the 27 items of the evaluation instrument. The second stage of the analysis considered the content and construct validity and reliability of the instrument. Next, a factor analysis was conducted to reduce a large number of variables to a smaller number of factors, and to describe the relationships among the observed variables. Findings – Fall 2008 The data for Fall 2008 consisted of 475 useable instruments among the 487 that were received (97.5%) from eight of the ten universities. Twelve responses were deleted because of incomplete evaluation and/or demographic data. Among useable responses, there were 178 male (37.5%) and 297 female respondents (62.5%). Ninety-seven percent of the respondents were in their sophomore (28.6%), junior (41.9%), or senior (26.7%) year of studies, and 94% expected to earn a grade of “B” or higher. Lastly, 66% of the respondents thought the survey was better than the survey sponsored by their uni