# Use of Student Evaluation of Teaching (SET) Survey to Evaluate Effectiveness of Teaching in a Leadership Course among Dental Students over Three Years

Authors: Muhammad Nazir; Asim Al-Ansari; Khalifa AlKhalifa; Balgis Gaffar; Jehan AlHumaid
Journal: The Scientific World Journal
 (2020)

---

## Abstract

Leadership courses are being increasingly integrated into dental curricula. The study aimed to assess the validity and reliability of student evaluation of teaching (SET) instrument among dental students and to evaluate the effectiveness of teaching in a new leadership course over a period of three years. This cross-sectional study was conducted on fourth-year undergraduate dental students (N = 260) who took a practice management course over three consecutive years from 2014 to 2016. A 29-item SET questionnaire was administered among students who were willing to participate in the study. Out of 260 students, 185 returned completed surveys and the response rate was 71.15%. Factor analysis (principal component analysis) showed the validity of four dimensions of the SET instrument. Total variance explained by four dimensions was 62.80%. Cronbach’s alpha for the instrument was 0.95 and each dimension had fairly high internal consistency (>0.80). Treating students with respect (94%), accepting different viewpoints of students (94.1%), being flexible/open-minded (92.5%), and preparedness in the course (91.9%) were the most common effective teaching traits. Over the period of three years, 16 items showed improvement in teaching and there was a significant improvement in four items (P<0.05). In conclusion, it was found that SET is a valid instrument to evaluate the effectiveness of teaching in nonclinical courses in dentistry. This instrument should be used longitudinally to compare the effectiveness of teaching.


---

## Body

## 1. Introduction

Ongoing technological innovation, greater diversity in treatment options, higher patient demands, and increased health care legislation are bringing challenges for dental professionals [1, 2]. Therefore, effective management of patients, staff, and interaction with colleagues and other health care professionals and stakeholders are becoming more difficult and complex for new dental graduates [3]. Acquiring knowledge and skills in clinical, biomedical, and behavioral sciences in dental programs are not enough to successfully run the business of a dental practice. The understanding and mastery of the principles of business including marketing, human resources, and financial management are needed for graduating dentists in today’s highly competitive job market [4]. Hence, the inclusion of a leadership course in dental curricula has been stressed by researchers and practicing dentists to prepare dental students to effectively work as dental practitioners, academicians, and community leaders [5, 6]. A recent study about curricula of practice management courses emphasized the need for improvement in course topics and use of the latest teaching methodologies to enhance the preparedness of graduates for dental practice [7].The American Dental Association underscores the need for leadership development for dentists and offers several educational opportunities through conferences, podcasts, webinars, and online continuing education activities [8]. The Commission on Dental Accreditation (CODA) has established six standards for dental education programs, and one of the standards is the educational program that includes practice management and health care system [9]. However, leadership competencies are not only required in dental practice management, but also in teaching and learning, research, organized dentistry, community mobilization, and public health [10]. According to the American Dental Education Association Survey of Dental School Seniors (2018), 62.5% of respondents believed that they were prepared in practice administration [11].In the USA, leadership training is provided by teaching a practice management course in about half of the dental schools; however, various community outreach initiatives and other courses also offer leadership skills in remaining schools [10]. It was reported that dental students expected leadership roles in dental practice (97%) and outreach volunteer opportunities (72%), and the vast majority of them considered leadership competencies important for dental professionals [12]. Literature also showed that about 80% of dental students considered mentorship as the most effective way of learning leadership [13].A dental practice is the most important place for demonstrating leadership skills, although participating in professional activities and community programs, and taking part in dental associations also provide avenues for dynamic leadership opportunities [14]. Most dental students (84%) from Harvard Dental School of Medicine reported that new leadership practice management course enhanced their interest in communication, leadership, business, team management, dental care access, and practice management [5].In Saudi Arabia, graduating dental students should be furnished with leadership skills to effectively play their roles in organized dentistry and the provision of dental care particularly to underserved communities. They should also assume a leadership role in their highly competitive and demanding professional careers. The courses, under the name of practice management or administration, vary widely in their content, methodology of delivery, number of credit hours, the academic year of the program, and the level of participation by instructors. For instance, a survey of academic deans about the practice management course showed that 22 practice management courses were taught to third and fourth-year students in 10 Canadian dental schools with 27 to 109 hours of teaching. These courses broadly focused on ethics, human resource management, and private dental practice management [15].However, focusing on the current and future needs of the dental profession in the Kingdom of Saudi Arabia, a comprehensive course, was developed keeping in mind the national and international course guidelines and standards. A series of in-depth discussions took place among the faculty members of the Dental Public Health Division. The course broadly covers topics on leadership, management of patients and dental team, appointment, inventory, record management systems, health care systems, marketing, and financial management to name a few. Individual and group written assignments, such as case studies, letter writing, SWOT analysis, and team assessment survey, were included in the course. The course also included two guest speaker presentations. Over the period of three years, new topics were included in the course each year based on the input from faculty members, guest speakers, students, and emerging evidence in the practice management. The new onecredit course was delivered to fourth-year students and completed in one semester.A well-designed practice management course is one of many ways to teach management and leadership knowledge and skills to dental students. The success of a course depends upon the effectiveness of teaching which is usually evaluated by students, peers, and the faculty, but student evaluation is the most commonly done at the level of university [16]. However, dental students’ evaluation of teaching (SET) through a validated survey is scant in the literature. Therefore, the study aimed to evaluate (1) the validity and reliability of the SET instrument by performing its psychometric analysis and (2) the effectiveness of teaching in a leadership course over a period of three years.


## 2. Materials and Methods

### 2.1. Study Design and Participants

This cross-sectional study was conducted on undergraduate fourth-year dental students at the College of Dentistry, Imam Abdulrahman Bin Faisal University, Dammam, Saudi Arabia. The college is an accredited public dental institute in the Eastern province of the country. The Bachelor of Dental Surgery (BDS) program includes students from the second year to six year. However, the first year of these students is spent in learning science subjects at the Preparatory College in the University. The study analyzed the data of fourth-year students who participated in practice management, nonclinical, course over a period of three years. In 2014, the course was introduced to students (N = 72) who provided their responses to students’ evaluations of teaching (SET) instrument at the end of the course. Next year, another group of fourth-year students (N = 94) attended the same course and their data were collected using the same instrument. The third batch of the fourth-year students (N = 94) took the course in 2016 and their responses were gathered at the end of the course using the same instrument. Therefore, the data of fourth-year students from three batches of consecutive students were collected for the study. A convenience sample was used for the study by inviting all the students in three batches of fourth-year class over three years (N = 72 + 94 + 94 = 260 students).


### 2.2. Data Collection Instrument

The students’ evaluations of teaching (SET) instruments are used to evaluate the effectiveness of teaching in higher education. These instruments are considered more important than peer evaluation and faculty self-reports [16]. Dodeen validated a SET survey for the students of humanities, sciences, business, IT, agriculture, engineering, and law colleges in the Middle East. The instrument is not subject matter specific and is neither too short nor too long [16]. Therefore, the current study administered the same SET survey among dental students in Saudi Arabia [16]. The questionnaire comprised 29 items distributed in five dimensions such as teachers’ knowledge and organization, clear explanation, grading and evaluation, teaching methods, and relationship with students. There are 7 items in the “Teachers’ Knowledge and Organization” domain, 6 items in the “Clear Explanation” domain, 6 items in the “Grading and Evaluation” domain, 4 items in the “Teaching Methods” domain, and 6 items in “Relationship with Students” domain. A 5-point Likert scale (from very poor = 1 to very good = 5) was used for each item of the instrument [16].


### 2.3. Procedures

A self-administered questionnaire in English was distributed among dental students. English is the medium of instruction in the College. Each year, the Vice Dean of Academic Affairs granted permission to distribute questionnaires. The copies of questionnaires were provided to all students in the classes at the end of the course during the last weeks of the first semester each year. One researcher briefed students about the survey, and they were encouraged to ask questions if any item needed further explanation. They were informed about no negative consequences in case students refusing to participate in the study. The confidentiality and privacy of students’ responses were maintained by administering an anonymous questionnaire. Those students who agreed to voluntary participation were included in the survey. The questionnaire administration was completed by one researcher within approximately 20 minutes. Forty-nine students in 2014, 72 students in 2015, and 64 students in 2016 returned completed questionnaires. Of 260 students who took the course over three years, 185 filled questionnaires, and the response rate was 71.15%.


### 2.4. Statistical Analysis

The data were entered in MS Excel (2010) and transferred to IBM SPSS Statistics for Windows, version 22 (IBM Corp., Armonk, N.Y., USA) for statistical analysis. Descriptive statistics included means, standard deviations, and frequency distributions. The psychometric analysis of the questionnaire was performed to evaluate its validity and reliability. Exploratory Factor Analysis (EFA) was performed to determine the construct validity of the questionnaire [17, 18]. The reliability of the questionnaire can be assessed by the internal consistency of the instrument [18]. The internal reliability of the instrument is measured by Cronbach’s alpha and 0.7 is its minimum recommended value. Internal reliability of each scale, as well as the correlation of items in the questionnaire, were computed [19]. Independent sample t-test was performed to report differences in the mean score of responses by male and female students. Kruskal-Wallis Test was performed to evaluate the effectiveness of teaching over three years. For a better interpretation of data, very poor and poor response were combined to present poor, and good and very good responses were combined to present good as mentioned above. A P value of 0.05 was used for statistical significance.


## 2.1. Study Design and Participants

This cross-sectional study was conducted on undergraduate fourth-year dental students at the College of Dentistry, Imam Abdulrahman Bin Faisal University, Dammam, Saudi Arabia. The college is an accredited public dental institute in the Eastern province of the country. The Bachelor of Dental Surgery (BDS) program includes students from the second year to six year. However, the first year of these students is spent in learning science subjects at the Preparatory College in the University. The study analyzed the data of fourth-year students who participated in practice management, nonclinical, course over a period of three years. In 2014, the course was introduced to students (N = 72) who provided their responses to students’ evaluations of teaching (SET) instrument at the end of the course. Next year, another group of fourth-year students (N = 94) attended the same course and their data were collected using the same instrument. The third batch of the fourth-year students (N = 94) took the course in 2016 and their responses were gathered at the end of the course using the same instrument. Therefore, the data of fourth-year students from three batches of consecutive students were collected for the study. A convenience sample was used for the study by inviting all the students in three batches of fourth-year class over three years (N = 72 + 94 + 94 = 260 students).


## 2.2. Data Collection Instrument

The students’ evaluations of teaching (SET) instruments are used to evaluate the effectiveness of teaching in higher education. These instruments are considered more important than peer evaluation and faculty self-reports [16]. Dodeen validated a SET survey for the students of humanities, sciences, business, IT, agriculture, engineering, and law colleges in the Middle East. The instrument is not subject matter specific and is neither too short nor too long [16]. Therefore, the current study administered the same SET survey among dental students in Saudi Arabia [16]. The questionnaire comprised 29 items distributed in five dimensions such as teachers’ knowledge and organization, clear explanation, grading and evaluation, teaching methods, and relationship with students. There are 7 items in the “Teachers’ Knowledge and Organization” domain, 6 items in the “Clear Explanation” domain, 6 items in the “Grading and Evaluation” domain, 4 items in the “Teaching Methods” domain, and 6 items in “Relationship with Students” domain. A 5-point Likert scale (from very poor = 1 to very good = 5) was used for each item of the instrument [16].


## 2.3. Procedures

A self-administered questionnaire in English was distributed among dental students. English is the medium of instruction in the College. Each year, the Vice Dean of Academic Affairs granted permission to distribute questionnaires. The copies of questionnaires were provided to all students in the classes at the end of the course during the last weeks of the first semester each year. One researcher briefed students about the survey, and they were encouraged to ask questions if any item needed further explanation. They were informed about no negative consequences in case students refusing to participate in the study. The confidentiality and privacy of students’ responses were maintained by administering an anonymous questionnaire. Those students who agreed to voluntary participation were included in the survey. The questionnaire administration was completed by one researcher within approximately 20 minutes. Forty-nine students in 2014, 72 students in 2015, and 64 students in 2016 returned completed questionnaires. Of 260 students who took the course over three years, 185 filled questionnaires, and the response rate was 71.15%.


## 2.4. Statistical Analysis

The data were entered in MS Excel (2010) and transferred to IBM SPSS Statistics for Windows, version 22 (IBM Corp., Armonk, N.Y., USA) for statistical analysis. Descriptive statistics included means, standard deviations, and frequency distributions. The psychometric analysis of the questionnaire was performed to evaluate its validity and reliability. Exploratory Factor Analysis (EFA) was performed to determine the construct validity of the questionnaire [17, 18]. The reliability of the questionnaire can be assessed by the internal consistency of the instrument [18]. The internal reliability of the instrument is measured by Cronbach’s alpha and 0.7 is its minimum recommended value. Internal reliability of each scale, as well as the correlation of items in the questionnaire, were computed [19]. Independent sample t-test was performed to report differences in the mean score of responses by male and female students. Kruskal-Wallis Test was performed to evaluate the effectiveness of teaching over three years. For a better interpretation of data, very poor and poor response were combined to present poor, and good and very good responses were combined to present good as mentioned above. A P value of 0.05 was used for statistical significance.


## 3. Results

The validity of the instrument was confirmed by the factor analysis shown in Table1. Kaiser’s criterion of Eigen values ≥ 1 and scree plot were used for factor extraction. Total variance explained by four scales was 62.80%. Preparation and assessment (Factor 1) accounted for 20.71% of the variance and interaction with students (Factor 2) represented 15.64% of the variance, while teaching and learning (Factor 3) and preparedness of instructor (Factor 4) explained 14% and 12.44% of the variance, respectively (Table 1). High internal consistency of the instrument and each scale confirmed the reliability of the instrument. Cronbach’s alpha for the instrument was 0.95 and each dimension had fairly high internal consistency (Cronbach’s for preparation and assessment = 0.93, interaction with students = 0.87, teaching and learning = 0.89, and preparedness of instructor = 0.80).Table 1
Factor analysis of students’ evaluations of teaching.
Factor 1 preparation and assessmentFactor 2 interaction with studentsFactor 3 teaching and learningFactor 4 preparedness of instructorClear presentations of course materials0.757Assignments, projects, activities, ... etc. are clear0.656Examples are used to simplify difficult points0.667Clear explanations of concepts and principles0.657Emphasize difficult points and facts0.659Examination questions are clear0.472Examination covers content emphasized by the instructor0.523Grading criteria are clear0.403Offer useful feedback on assignments, projects, activities0.575Offer useful feedback on tests0.678Instructor’s grading policy is fair0.585Use variety of assessment methods0.534Available outside the classroom for assisting students0.506Cares for students’ learning0.663Treat students with respect0.719Accept different viewpoints presented by students0.733Treat all students fairly0.728Flexible/open-minded when dealing with students0.708Effective use of class time0.679The class time is carefully planned0.761Effective classroom management0.574Use teaching aids and technology effectively0.597Present course materials at an appropriately paced sequence0.535Encourage students to seek knowledge from multiple resources0.647Motivate students to learn0.628The instructors are well-prepared in their course0.593The instructors are informative when responding to students’ questions0.631The instructors state goals and objectives clearly0.518Lectures are well organized0.615Eigenvalue13.761.921.361.15% of total variance20.7115.641412.44Total variance62.80Table2 shows students’ responses about the effectiveness of teaching during the last three years. It can be seen that most students reported that treating students with respect (94%), accepting different viewpoints presented by students (94.1%), being flexible/open-minded when dealing with students (92.5%), and preparedness (91.9%) in the course were good or very good. No statistically significant differences were observed between male and female students’ responses.Table 2
Students’ responses about the effectiveness of teaching.
Assessments and evaluationPoor (%)Moderate (%)Good (%)Examination questions are clear4.921.174.1Examination covers content emphasized by the instructor118.480.5Grading criteria are clear3.715.780.6Clear presentations of course materials4.320.075.7Assignments, projects, activities, ...etc. are clear10.334.655.1Examples are used to simplify difficult points6.519.574.1Clear explanations of concepts and principles3.824.971.4Emphasize difficult points and facts5.930.363.7Offer useful feedback on assignments, projects, activities11.425.463.2Offer useful feedback on tests618.975.2Instructor’s grading policy is fair2.713.084.3Use variety of assessment methods1.618.979.5Available outside the classroom for assisting students2.113.584.3Interaction with studentsCare for students’ learning2.19.288.7Treat students with respect2.23.894Accept different viewpoints presented by students14.994.1Treat all students fairly4.910.884.3Flexible/open-minded when dealing with students1.65.992.5Teaching and learningEffective use of class time15.136.248.7The class time is carefully planned12.934.652.4Effective classroom management11.924.963.2Use teaching aids and technology effectively8.131.460.5Present course materials at an appropriately paced sequence3.323.273.5Encourage students to seek knowledge from multiple resources5.424.370.3Motivate students to learn5.422.771.9Preparedness of instructorThe instructors are well-prepared in their course0.57.691.9The instructors are informative when responding to students’ questions2.28.689.2The instructors state goals and objectives clearly2.211.486.5Lectures are well organized2.713.583.8The comparison of students’ responses over three years showed that there was improvement in more than half of the items of effective teaching however, statistically significant improvement was seen in four items (P<0.05) (Table 3).Table 3
Students’ responses about the effectiveness of teaching over last three years.
Assessments and evaluationYear 2014/15Year 2015/16Year 2016/17P valueMeanMeanMeanExamination questions are clear4.044.013.810.003∗Examination covers content emphasized by the instructor4.244.144.000.143Grading criteria are clear4.274.114.210.632Clear presentations of course materials4.043.984.230.718Assignments, projects, activities, ... etc. are clear3.843.473.810.105Examples are used to simplify difficult points3.943.884.230.107Clear explanations of concepts and principles3.983.884.140.897Emphasize difficult points and facts3.613.814.140.036∗Offer useful feedback on assignments, projects, activities3.763.664.020.195Offer useful feedback on tests3.534.224.160.001∗Instructor’s grading policy is fair4.124.264.260.511Use variety of assessment methods3.884.184.190.044∗Available outside the classroom for assisting students4.144.264.400.118Interaction with studentsCare for students’ learning4.334.304.490.231Treat students with respect4.594.474.720.546Accept different viewpoints presented by students4.494.484.510.879Treat all students fairly4.554.174.330.166Flexible/open-minded when dealing with students4.694.394.560.021∗Teaching and learningEffective use of class time3.513.403.840.950The class time is carefully planned3.673.483.670.366Effective classroom management3.863.633.950.315Use teaching aids and technology effectively3.593.723.910.445Present course materials at an appropriately paced sequence3.923.884.120.268Encourage students to seek knowledge from multiple resources3.943.824.230.024∗Motivate students to learn4.043.884.230.749Preparedness of instructorThe instructors are well-prepared in their course4.534.384.490.350The instructors are informative when responding to students’ questions4.514.274.440.405The instructors state goals and objectives clearly4.314.204.440.411Lectures are well organized4.144.174.230.601


## 4. Discussion

The SET instrument is a multidimensional, reliable, stable, and valid tool and is used frequently to obtain diagnostic feedback on teaching by instructors. Other purposes of collecting SETs include course selection by students and decision-making about staff’s teaching effectiveness by administrators [20]. Our study evaluated the psychometric properties of a SET instrument among dental students in a leadership, nonclinical, course. This revalidation of the instrument has provided valuable information and it can be used to obtain a comprehensive and multidimensional assessment of the effectiveness of teaching in nonclinical undergraduate courses in dentistry. In our study, construct validity was confirmed by the factor analysis that grouped items into four dimensions/scales and the instrument showed high internal consistency (Cronbach’s alpha 0.95).Revalidation of the SET instrument should be based on a theoretical model in teaching or empirical testing that involves procedures to ensure its validity and reliability [21]. The present study evaluated the validity of the instrument by performing factor analysis. The original instrument from the previous study consisted of 29 items in five dimensions [16]. In contrast, our analysis of data organized the items into four dimensions containing varying number of items in each dimension compared to the previous study [16]. However, similar to our findings, Gurosy and Umbreit reported four dimensions of the SET instruments [22]. It is also known that most SET instruments broadly contain three dimensions, namely, organization, student grading, and instructor-student interaction [16]. These discrepancies in the content and construction of different SET instruments could be related to varying expectations and needs of various institutions [23]. Moreover, these variations could have resulted because of the number and types of participants and methodological differences including data analysis techniques.When students’ responses about different dimensions of the instrument were analyzed then instructor-student relationship and instructor’s preparedness (organization) stood out as ≥90% of students claimed that instructors treated students with respect, accepted different viewpoints, demonstrated flexibility, and were prepared in the course. Schonwetter et al. [24] found that 30% (n = 102) of students in classroom teaching reported the organization and 27% (n = 91) interaction of teacher with students/individual rapport with students were the most common characteristics of effective teachers while only 4% considered assessment an element of effectiveness of teaching. Similarly, Jahangiri and Mucciolo identified development, organization, and design of content (>50%) and interaction with students (34.6%) and caring attitudes (28.8%) as the most predominant features of teacher’s effectiveness in nonclinical programs [25]. In a study by McAndrew et al. students reported that student-teacher interaction/character (50%) was the most common theme of characteristics of effective teachers, followed by competence/preparedness of instructor theme (34%) and communication (17%) [26].Student evaluation of teaching is widely used to assess the effectiveness of teaching in higher education globally. It is also documented that SETs can be used to obtain students’ feedback to monitor the quality of teaching to serve the purpose of quality assurance management in universities [27]. In addition, it shows the promise of the institution to monitor teaching performance as part of continuous improvement activities in the program [28]. It has been suggested that meaningful inferences can be drawn from the results of evaluations that are conducted repeatedly at regular intervals to ensure the improvement process in teaching. Moreover, these evaluations should be combined with the testing of data [29].The improvement in teaching observed in successive years was reported in 8 out of 14 studies involving longitudinal comparisons of students’ evaluations [30]. In the present study, student feedback was observed longitudinally over a period of three years to report the quality of teaching provided in the course. Results obtained through the analysis of data showed that 16 of 29 items of the SET instrument demonstrated improvement in teaching from 2014 to 2016. Overall, the mean score of these 16 items increased, but significant increase was observed in four items: “encourage students to seek knowledge from multiple resources” (P=0.024), “offer useful feedback on tests” (P=0.001), and “emphasize difficult points and facts” (P=0.036). On the other hand, the mean score in two items such as “examination questions are clear” (P=0.003) and “flexible/open-minded when dealing with students” (P=0.021) significantly decreased thus requiring improvement in these areas. These results support the common notion that improvement in teaching can be obtained by using student evaluations.There are claims about the positive and valuable influence of student evaluation of teaching on academic standards [30]. There are also concerns and confusion about student evaluation. It has been argued the student rating of effective teaching can provide inaccurate inferences leading to unfair decision including enhancing student grades and negatively affecting standards of teaching [30, 31]. However, the literature lacks convincing evidence about the impact of student evaluation on inflating student grades, lowering course requirement, and compromising teaching standards [30].As discussed previously, student evaluation of teaching has benefits of improving the effectiveness of teaching, academic standards, and quality assurance management. It is, therefore, suggested that student evaluations should be combined with self-evaluation and peer assessment because student rating alone is an inadequate measure of teaching effectiveness as students have limitations of assessing the appropriateness of course material and assessment methods [31–33]. Therefore, in the present study, student evaluations in addition to the feedback of faculty members and guest speakers and self-assessment were used to improve the quality of the course by modifying its content, delivery, and assessment.Nonparticipation (28.85%) in the present study was mainly because of the absence of some students on the day of questionnaire administration. The college offers an intense undergraduate dentistry program; hence some students tend to avoid attending the last session of certain courses so that they can better prepare for their final examinations. A large data set obtained from a bigger sample of students could have better captured the study outcomes. Regardless of this limitation, this study provided an opportunity to guide best practices in teaching nonclinical courses in dentistry.


## 5. Conclusions

Based on the study findings, it is concluded that SET, a valid and reliable instrument, can be used to evaluate the effectiveness of teaching in nonclinical courses in dentistry. Longitudinal comparison of student’s evaluations over the years can contribute to improving teaching. It is emphasized that a validated SET instrument should be used as part of evidence-based practices in dental education to evaluate the effectiveness of teaching. Moreover, data should be monitored longitudinally over consecutive years particularly to identify areas needing improvement and to maintain elements of effective teaching in the course. Dental institutions, educational policymakers, and instructors should ensure that the course content, teaching methods, assessment instruments are continuously updated using student evaluation in conjunction with self-evaluation and peers’ observations.



---

*Source: 6436102-2020-06-01.xml*