Concept Inventories as Predictors of Changing Pre-Engineering Skills

This article was motivated by a need to understand the academic strengths and weaknesses of engineering students entering first-year. The underlying impetus was the improvement of the educational process. At our institution, and perhaps in general, there is an anecdotal hypothesis that the mathematical skills, or more generally, the academic strengths of first-year engineering students have been significantly declining in the last two decades. To address this hypothesis, a longitudinal study of pre-engineering indicators for first year engineering students in a Canadian university transfer program was initiated almost a decade ago through the use of two assessment exams and by collection and consolidation of high school achievement data. It is not unusual to use these academic indicators as a measure of skill level. To this end, two assessment tools (Math Advisory and Engineering Assessment) were used to generate data with which to test this hypothesis. Simple statistical analyses and temporal trend plots were used to examine the data. The results of the data analysis gave rise to some interesting observations. The Math Advisory exam mean scores for engineering students considered in this study were consistently between 50-55% during the period considered, with little variation from year to year. The same trend was observed for the Engineering Assessment mean scores (Force Concept Inventory). In both cases, the changes in the mean scores over time were analyzed using a t-test, and the analysis indicated that the changes in the mean scores were not statistically significant. Overall high school achievement of these students entering first-year also showed little variation during this same time period. Comparison of the assessment exam mean scores to high school achievement data showed that performance on assessment exams was much poorer than high school achievement might have predicted. There was ~30% reduction in performance when the assessment indicators were compared to high school achievement. As a result of the overall poor performance on the assessment exams (50-55%), as well as inconsistency between results on the Math Advisory exam and high school achievement in math, the engineering program has regularly implemented weekly math and engineering tutorial sessions to supplement the course curricula. However, at this stage the success of these initiatives has not been verified. Although the data at first glance (10.89 compared to 10.6 for Math Advisory exam) might indicate a decline in the performance of engineering students on the math assessment exam, the data does not support the anecdotal hypothesis nor the quantitative evidence of markedly declining math skills in first-year engineering students at our institution, at least during the time period considered. Assessment tools are often used in a predictive way to gauge the overall skills of engineering students. They are also useful in setting engineering program directives. It is clear from this article that the academic averages obtained in high school, may not necessarily reflect the skill level of the students entering first-year, especially in mathematics. A further analysis of these assessment results in light of first-year academic achievement might reveal further information regarding the success or lack thereof in delivering the first year curriculum.