The Effect of Homework on Exam Performance: Experimental Results from Principles of Economics

[Author Affiliation]Steve Trost, , , , strost@vt.eduDjavad Salehi-Isfahani, , , , salahi@vt.edu[Acknowledgment]Thanks to Chris Makler and the folks at Aplia for being very generous with their time and information; to Paul Romer, who initially suggested the idea for this project several years ago; and to anonymous readers, whose insightful comments improved this article.1. IntroductionOver the past decade, the use of online homework systems in teaching of "Principles of Economics" courses has increased sharply. Enrollment in Aplia, the largest of these systems, has increased from 25,000 students in 2003 to over 286,000 students in 2011.1 Proponents of these systems argue that they make it possible to assign meaningful homework to students in large lecture classes without imposing an undue grading burden on the instructor or teaching assistants. While these online homework programs may not be able to replicate the complexity of some traditional pencil-and-paper assignments, they can engage the students and require them to think about the topics outside of the lecture hall. As with any new teaching or learning tool, the question of efficacy is an important one. Does the completion of online homework assignments have a significant effect on a student's exam performance?We try to answer this question by taking advantage of a unique experimental setup using the Aplia online homework system. A serious problem faced by many randomized experiments of homework assignment is that it is not possible to randomly assign students to classes or (from a fairness standpoint) to prevent some students in a class from doing a homework assignment. Previous studies have thus exploited differences in exam performance between classes with and without homework--an approach that creates difficulties with interpreting the results since different classes, even when taught by the same instructor, may differ in instructor effectiveness and mix of students because of the time or location of the class (see Lass, Morzuch, and Rogers 2007; Emerson and Mencken 2011). We try to solve this problem by randomizing within the same class but on different assignments. For three topics in a Principles of Microeconomics course (supply and demand, elasticity, and consumer behavior), we assign students to a treatment group for whom the assignment for that topic is optional (will not count toward the course grade) and a control group for whom the assignment is required. By allowing students to skip homework assignments pertaining to particular topics with no grade penalty, we are able to more precisely test the effect of homework completion on performance on exam questions related to each topic. Thus, instead of measuring the overall effect of homework on exam performance, we are measuring how topic-specific homework completion affects topic-specific exam performance.The random assignment does not exactly predict who does and does not do a particular assignment because the more motivated students are more likely to complete optional homework assignments. However, since assignment is highly correlated with completion, the experiment produces a strong instrument that allows us to identify the effect of completion on exam grade. We take advantage of this correlation and use an instrumental variables (IV) estimation technique similar to that used by Angrist (1990) and Bloom et al. (1997). Results from the IV analysis are compared to results from standard ordinary least squares (OLS) estimations that use homework completion or assignment as explanatory variables. We also perform OLS on a specific subset of the population studied, for whom homework assignment and completion are perfectly correlated. In each specification, our results rely heavily on a unique variable called the "zero topic" score, which is simply the student's score on exam questions related to homework assignments that the entire class was required to do. …