ON THE ASSESSMENT OF RETENTION EFFECTS IN EDUCATIONAL EXPERIMENTS

THE FIRST part of this paper is essentially a plea for a greater emphasis on long-term follow up measures in studies of the effects of instruction al treatments and for the investigation of instruc tional treatments specifically designed to facilitate such long-term effects. Some methodological prob lems of investigating such effects will also be con sidered. The second half of the paper is devoted to a consideration of several procedures for studying long-term retention. For about a year now the writer has been con cerned with the problem of designing experiments to compare the relative effectiveness of different strategies of instruction in the context of computer assisted instruction. For example, researchers at the Computer Assisted Instruction Laboratory at at Penn State have been concerned with the relative effectiveness of branching versus nonbranching in structional programs, feedback versus nonfeed back, programs employing visual and auditory stim ulus materials, and other similar problems. In considering these research problems it was hypoth esized that some instructional variations might be of little value in facilitating the amount learned o r the speed of learning, but might have their primary effects on long-term retention. This hypothesis suggests that some instructional treatments m ig ht have effects on learning and retention similar to that of the variable-ratio reinforcement schedule of the experimental literature. An instructional treat ment might produce relatively inefficient learning, but greater resistance to forgetting than some other methods. For example, one might hypothesize that the act of making an overt response in an au to-in structional program will not facilitate immediate re tention, but will have a positive effect on long-term retention. If such a hypothesis was correct fora given treatment comparison, astudy which em ployed only one measure of learning immediately fol lowing the treatment would result in non-significant differences and the misleading conclusion that one method was no better than the other. Significant dif ferences attributable to the instructional variation would only be observed on a delayed retention meas ure. Yvonne Brackbill and her colleagues (3) have obtained such results in an experiment comparing the effects of immediate versus delayed feedback. Brackbill found no difference between the treatments on the measure of learning, but did obtain signifi cant differences favoring a ten second feedback de lay for one-day and seven-day retention measures. The possibility that other instructional treatments may have similar effects is a good argument for placing greater emphasis on follow-up retention measures in educational experiments. Following the lead of Brackbill and other writers who have urged the inclusion of delayed retention measures in ex periments on instruction, several research designs were planned which included measures of delayed retention. What seemed initially to be a fairly sim ple problem of design and analysis turned out to be much more complicated. The purpose of the pres ent paper is to describe some of the difficulties in assessing the effects of an instructional treatment on retention over time, and to suggest several pro cedures which may be useful in such an experi ment .2 One common research design used to study the effects of different instructional treatments on learn ing and retention is a modified form of the pre-post test design. In this design subjects are assigned at random to two or more experimental treatments. Prior to beginning the experimental treatment the subjects may be pretested on the subject matter to be taught. The experimental treatments are then administered followed by one or more alternate forms of an achievement test to measure the degree of learning. If the investigator is concerned only with learning or immediate achievement, one post test measure may be given immediately following the treatment; however, if the investigator is con cerned with retention over time, one or more addi tional follow-up tests may be administered atsuc cessive intervals following training. Owing to the difficulties of reassembling subjects for follow-up testing, in actual practice most investigators obtain