An Automatic Course Assessment Tool for Weakness Identification

We develop an automatic course assessment tool that gives numerical indicators on fulfilling course learning outcomes and identifies course weaknesses for future improvement. Four years of application indicate that this tool is very effective in revealing course weaknesses and in reducing the time spent on course assessment. Course assessment is an important requirement for accreditation but is very time-consuming. Besides teaching the courses in classroom or online, faculty have to spent significant time to collect and analyze student data to meet program assessment and accreditation requirements. While many accreditation agents have their requirements on course assessment and continuous improvement, they usually gave little instruction on how to collect assessment samples, analyze student data and find areas for future improvement. In the literature, there are very few quantitative methods that can be used for automatic, large-scale assessment. In our recent preparation for ABET accreditation (1), we follow the Faculty Course Assessment Report (FCAR) process (2), which is also adopted by many other universities (3, 4, 5), to assess our program and courses. At the beginning of the semester, the course instructor receives clarifications and training and agrees on the FCAR completion date (2 weeks after the semester ends). The instructor keeps record of evaluation data during the semester, and writes an FCAR to assess the course and to identify existing problems. In the beginning of the following semester, the computer science faculty analyzes the previous semester FCARs and agrees on a set of actions. When the course is offered next time, the instructor who teaches this course will review the FCAR and take the agreed actions in the course to improve it. The core of the assessment in FCAR is to use two vectors: EEMU (Excellent, Effective, Minimal, and Unsatisfactory) and EPAN (Exemplary, Proficient, Apprentice, and Novice) to determine whether each course learning outcome or program outcome is successfully met. None of the references specifies the method to determine a student's proficiency in EEMU rating and ability in EPAN rating. In our earlier assessment cycles, many instructors had to check each student's performance in the homework, projects and tests to make a subjective estimate to determine whether the student achievement in the course learning outcome is excellent, effective, minimal, or unsatisfactory; or exemplary, proficient, apprentice, or novice in the program learning outcome. Such comprehensive and subjective estimation was ambiguous, time-consuming and yielded inaccurate and inconsistent results. Some instructors estimated the students' achievement from their acquaintance with the students; others said they had to come up with numbers that might be inaccurate. A more severe problem emerged when we compared the assessment results across different courses. Since each instructor estimated his or her EEMU and EPAN vectors differently, the results from different courses were completely inconsistent and incomparable. Through comparison and experiment, we developed a selective and objective method that calculates the assessment results quantitatively. We designed a set of spreadsheets to collect student performance data, to calculate the assessment results automatically, and to identify areas for future improvement. In this paper, we share our selective and objective assessment method and the way to automatically identify weaknesses in the course in Section 2 and 3. The Excel spreadsheets are introduced in Section 4. Even though this method is designed for ABET accreditation of our computer science program, it can be used in accreditations and assessment for all disciplines. II. ASSESSMENT COMPUTATION After using the FCAR procedures for one semester, our faculty found that the comprehensive and subjective method to determine the EEMU and EPAN vectors is time-consuming, inaccurate and inconsistent, and will be indefensible in future ABET interviews. The CS faculty asked for an accurate but less time-consuming method. So, we decided to design a selective and objective method for calculating the EEMU and EPAN vectors. In this method, a small set (usually three to four) of core and typical questions for each learning outcome is selected to conduct the assessment, and student scores are used to determine these EEMU and EPAN vectors. We also think that the vectors should reflect the proficiency and ability of students who passed the course, not those who failed and need to repeat. In all courses in our CS program, a student has to get